{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.4 BERTを用いたレビュー文章に対する感情分析モデルの実装と学習・推論\n",
    "\n",
    "本ファイルでは、BERTを使用し、IMDbデータのポジ・ネガを分類するモデルを学習させ、推論します。また推論時のSelf-Attentionを可視化します。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "※　本章のファイルはすべてUbuntuでの動作を前提としています。Windowsなど文字コードが違う環境での動作にはご注意下さい。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.4 学習目標\n",
    "\n",
    "1.\tBERTのボキャブラリーをtorchtextで使用する実装方法を理解する\n",
    "2.\tBERTに分類タスク用のアダプターモジュールを追加し、感情分析を実施するモデルを実装できる\n",
    "3.\tBERTをファインチューニングして、モデルを学習できる\n",
    "4.  BERTのSelf-Attentionの重みを可視化し、推論の説明を試みることができる\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 事前準備\n",
    "\n",
    "- 書籍の指示に従い、本章で使用するデータを用意します"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "import torch \n",
    "from torch import nn\n",
    "import torch.optim as optim\n",
    "import torchtext\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 乱数のシードを設定\n",
    "torch.manual_seed(1234)\n",
    "np.random.seed(1234)\n",
    "random.seed(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDbデータを読み込み、DataLoaderを作成（BERTのTokenizerを使用）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 前処理と単語分割をまとめた関数を作成\n",
    "import re\n",
    "import string\n",
    "from utils.bert import BertTokenizer\n",
    "# フォルダ「utils」のbert.pyより\n",
    "\n",
    "\n",
    "def preprocessing_text(text):\n",
    "    '''IMDbの前処理'''\n",
    "    # 改行コードを消去\n",
    "    text = re.sub('<br />', '', text)\n",
    "\n",
    "    # カンマ、ピリオド以外の記号をスペースに置換\n",
    "    for p in string.punctuation:\n",
    "        if (p == \".\") or (p == \",\"):\n",
    "            continue\n",
    "        else:\n",
    "            text = text.replace(p, \" \")\n",
    "\n",
    "    # ピリオドなどの前後にはスペースを入れておく\n",
    "    text = text.replace(\".\", \" . \")\n",
    "    text = text.replace(\",\", \" , \")\n",
    "    return text\n",
    "\n",
    "\n",
    "# 単語分割用のTokenizerを用意\n",
    "tokenizer_bert = BertTokenizer(\n",
    "    vocab_file=\"./vocab/bert-base-uncased-vocab.txt\", do_lower_case=True)\n",
    "\n",
    "\n",
    "# 前処理と単語分割をまとめた関数を定義\n",
    "# 単語分割の関数を渡すので、tokenizer_bertではなく、tokenizer_bert.tokenizeを渡す点に注意\n",
    "def tokenizer_with_preprocessing(text, tokenizer=tokenizer_bert.tokenize):\n",
    "    text = preprocessing_text(text)\n",
    "    ret = tokenizer(text)  # tokenizer_bert\n",
    "    return ret\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# データを読み込んだときに、読み込んだ内容に対して行う処理を定義します\n",
    "max_length = 256\n",
    "\n",
    "TEXT = torchtext.data.Field(sequential=True, tokenize=tokenizer_with_preprocessing, use_vocab=True,\n",
    "                            lower=True, include_lengths=True, batch_first=True, fix_length=max_length, init_token=\"[CLS]\", eos_token=\"[SEP]\", pad_token='[PAD]', unk_token='[UNK]')\n",
    "LABEL = torchtext.data.Field(sequential=False, use_vocab=False)\n",
    "\n",
    "# (注釈)：各引数を再確認\n",
    "# sequential: データの長さが可変か？文章は長さがいろいろなのでTrue.ラベルはFalse\n",
    "# tokenize: 文章を読み込んだときに、前処理や単語分割をするための関数を定義\n",
    "# use_vocab：単語をボキャブラリーに追加するかどうか\n",
    "# lower：アルファベットがあったときに小文字に変換するかどうか\n",
    "# include_length: 文章の単語数のデータを保持するか\n",
    "# batch_first：ミニバッチの次元を先頭に用意するかどうか\n",
    "# fix_length：全部の文章を指定した長さと同じになるように、paddingします\n",
    "# init_token, eos_token, pad_token, unk_token：文頭、文末、padding、未知語に対して、どんな単語を与えるかを指定\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# フォルダ「data」から各tsvファイルを読み込みます\n",
    "# BERT用で処理するので、10分弱時間がかかります\n",
    "train_val_ds, test_ds = torchtext.data.TabularDataset.splits(\n",
    "    path='./data/', train='IMDb_train.tsv',\n",
    "    test='IMDb_test.tsv', format='tsv',\n",
    "    fields=[('Text', TEXT), ('Label', LABEL)])\n",
    "\n",
    "# torchtext.data.Datasetのsplit関数で訓練データとvalidationデータを分ける\n",
    "train_ds, val_ds = train_val_ds.split(\n",
    "    split_ratio=0.8, random_state=random.seed(1234))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BERTはBERTが持つ全単語でBertEmbeddingモジュールを作成しているので、ボキャブラリーとしては全単語を使用します\n",
    "# そのため訓練データからボキャブラリーは作成しません\n",
    "\n",
    "# まずBERT用の単語辞書を辞書型変数に用意します\n",
    "from utils.bert import BertTokenizer, load_vocab\n",
    "\n",
    "vocab_bert, ids_to_tokens_bert = load_vocab(\n",
    "    vocab_file=\"./vocab/bert-base-uncased-vocab.txt\")\n",
    "\n",
    "\n",
    "# このまま、TEXT.vocab.stoi= vocab_bert (stoiはstring_to_IDで、単語からIDへの辞書)としたいですが、\n",
    "# 一度bulild_vocabを実行しないとTEXTオブジェクトがvocabのメンバ変数をもってくれないです。\n",
    "# （'Field' object has no attribute 'vocab' というエラーをはきます）\n",
    "\n",
    "# 1度適当にbuild_vocabでボキャブラリーを作成してから、BERTのボキャブラリーを上書きします\n",
    "TEXT.build_vocab(train_ds, min_freq=1)\n",
    "TEXT.vocab.stoi = vocab_bert\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DataLoaderを作成します（torchtextの文脈では単純にiteraterと呼ばれています）\n",
    "batch_size = 32  # BERTでは16、32あたりを使用する\n",
    "\n",
    "train_dl = torchtext.data.Iterator(\n",
    "    train_ds, batch_size=batch_size, train=True)\n",
    "\n",
    "val_dl = torchtext.data.Iterator(\n",
    "    val_ds, batch_size=batch_size, train=False, sort=False)\n",
    "\n",
    "test_dl = torchtext.data.Iterator(\n",
    "    test_ds, batch_size=batch_size, train=False, sort=False)\n",
    "\n",
    "# 辞書オブジェクトにまとめる\n",
    "dataloaders_dict = {\"train\": train_dl, \"val\": val_dl}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([[ 101, 2023, 3185,  ...,    0,    0,    0],\n",
      "        [ 101, 2043, 1045,  ...,    0,    0,    0],\n",
      "        [ 101, 1045, 2066,  ...,    0,    0,    0],\n",
      "        ...,\n",
      "        [ 101, 1045, 2876,  ..., 1997, 1996,  102],\n",
      "        [ 101, 2023, 2003,  ...,    0,    0,    0],\n",
      "        [ 101, 1045, 2293,  ...,    0,    0,    0]]), tensor([128, 158, 178, 150, 112, 142, 248,  90, 256, 176, 224, 250, 256, 142,\n",
      "        256, 200, 199, 185, 256, 256, 122, 184,  97, 148, 161, 211, 256, 256,\n",
      "         57, 256, 146, 129]))\n",
      "tensor([1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0,\n",
      "        0, 1, 1, 1, 0, 1, 1, 1])\n"
     ]
    }
   ],
   "source": [
    "# 動作確認 検証データのデータセットで確認\n",
    "batch = next(iter(val_dl))\n",
    "print(batch.Text)\n",
    "print(batch.Label)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'when', 'i', 'saw', 'this', 'movie', ',', 'i', 'was', 'amazed', 'that', 'it', 'was', 'only', 'a', 'tv', 'movie', '.', 'i', 'think', 'this', 'movie', 'should', 'have', 'been', 'in', 'theaters', '.', 'i', 'have', 'seen', 'many', 'movies', 'that', 'are', 'about', 'rape', ',', 'but', 'this', 'one', 'stands', 'out', '.', 'this', 'movie', 'has', 'a', 'kind', 'of', 'realism', 'that', 'is', 'very', 'rarely', 'found', 'in', 'movies', 'today', ',', 'let', 'alone', 'tv', 'movies', '.', 'it', 'tells', 'a', 'story', 'that', 'i', 'm', 'sure', 'is', 'very', 'realistic', 'to', 'many', 'rape', 'victims', 'in', 'small', 'towns', 'today', ',', 'and', 'i', 'found', 'it', 'to', 'be', 'very', 'bel', '##ie', '##vable', 'which', 'is', 'something', 'hard', 'to', 'find', 'in', 'other', 'rape', 'centered', 'movies', '.', 'i', 'also', 'thought', 'that', 'ti', '##ffa', '##ni', 'the', '##isse', '##n', 'and', 'brian', 'austin', 'green', 'were', 'awesome', 'in', 'the', 'parts', 'that', 'they', 'played', '.', 'i', 'definitely', 'recommend', 'this', 'movie', 'to', 'anyone', 'who', 'enjoys', 'movies', 'that', 'have', 'a', 'bit', 'of', 'a', 'harsh', 'reality', 'to', 'them', '.', 'i', 'enjoyed', 'it', 'very', 'much', '.', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n"
     ]
    }
   ],
   "source": [
    "# ミニバッチの1文目を確認してみる\n",
    "text_minibatch_1 = (batch.Text[0][1]).numpy()\n",
    "\n",
    "# IDを単語に戻す\n",
    "text = tokenizer_bert.convert_ids_to_tokens(text_minibatch_1)\n",
    "\n",
    "print(text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 感情分析用のBERTモデルを構築"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bert.embeddings.word_embeddings.weight→embeddings.word_embeddings.weight\n",
      "bert.embeddings.position_embeddings.weight→embeddings.position_embeddings.weight\n",
      "bert.embeddings.token_type_embeddings.weight→embeddings.token_type_embeddings.weight\n",
      "bert.embeddings.LayerNorm.gamma→embeddings.LayerNorm.gamma\n",
      "bert.embeddings.LayerNorm.beta→embeddings.LayerNorm.beta\n",
      "bert.encoder.layer.0.attention.self.query.weight→encoder.layer.0.attention.selfattn.query.weight\n",
      "bert.encoder.layer.0.attention.self.query.bias→encoder.layer.0.attention.selfattn.query.bias\n",
      "bert.encoder.layer.0.attention.self.key.weight→encoder.layer.0.attention.selfattn.key.weight\n",
      "bert.encoder.layer.0.attention.self.key.bias→encoder.layer.0.attention.selfattn.key.bias\n",
      "bert.encoder.layer.0.attention.self.value.weight→encoder.layer.0.attention.selfattn.value.weight\n",
      "bert.encoder.layer.0.attention.self.value.bias→encoder.layer.0.attention.selfattn.value.bias\n",
      "bert.encoder.layer.0.attention.output.dense.weight→encoder.layer.0.attention.output.dense.weight\n",
      "bert.encoder.layer.0.attention.output.dense.bias→encoder.layer.0.attention.output.dense.bias\n",
      "bert.encoder.layer.0.attention.output.LayerNorm.gamma→encoder.layer.0.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.0.attention.output.LayerNorm.beta→encoder.layer.0.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.0.intermediate.dense.weight→encoder.layer.0.intermediate.dense.weight\n",
      "bert.encoder.layer.0.intermediate.dense.bias→encoder.layer.0.intermediate.dense.bias\n",
      "bert.encoder.layer.0.output.dense.weight→encoder.layer.0.output.dense.weight\n",
      "bert.encoder.layer.0.output.dense.bias→encoder.layer.0.output.dense.bias\n",
      "bert.encoder.layer.0.output.LayerNorm.gamma→encoder.layer.0.output.LayerNorm.gamma\n",
      "bert.encoder.layer.0.output.LayerNorm.beta→encoder.layer.0.output.LayerNorm.beta\n",
      "bert.encoder.layer.1.attention.self.query.weight→encoder.layer.1.attention.selfattn.query.weight\n",
      "bert.encoder.layer.1.attention.self.query.bias→encoder.layer.1.attention.selfattn.query.bias\n",
      "bert.encoder.layer.1.attention.self.key.weight→encoder.layer.1.attention.selfattn.key.weight\n",
      "bert.encoder.layer.1.attention.self.key.bias→encoder.layer.1.attention.selfattn.key.bias\n",
      "bert.encoder.layer.1.attention.self.value.weight→encoder.layer.1.attention.selfattn.value.weight\n",
      "bert.encoder.layer.1.attention.self.value.bias→encoder.layer.1.attention.selfattn.value.bias\n",
      "bert.encoder.layer.1.attention.output.dense.weight→encoder.layer.1.attention.output.dense.weight\n",
      "bert.encoder.layer.1.attention.output.dense.bias→encoder.layer.1.attention.output.dense.bias\n",
      "bert.encoder.layer.1.attention.output.LayerNorm.gamma→encoder.layer.1.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.1.attention.output.LayerNorm.beta→encoder.layer.1.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.1.intermediate.dense.weight→encoder.layer.1.intermediate.dense.weight\n",
      "bert.encoder.layer.1.intermediate.dense.bias→encoder.layer.1.intermediate.dense.bias\n",
      "bert.encoder.layer.1.output.dense.weight→encoder.layer.1.output.dense.weight\n",
      "bert.encoder.layer.1.output.dense.bias→encoder.layer.1.output.dense.bias\n",
      "bert.encoder.layer.1.output.LayerNorm.gamma→encoder.layer.1.output.LayerNorm.gamma\n",
      "bert.encoder.layer.1.output.LayerNorm.beta→encoder.layer.1.output.LayerNorm.beta\n",
      "bert.encoder.layer.2.attention.self.query.weight→encoder.layer.2.attention.selfattn.query.weight\n",
      "bert.encoder.layer.2.attention.self.query.bias→encoder.layer.2.attention.selfattn.query.bias\n",
      "bert.encoder.layer.2.attention.self.key.weight→encoder.layer.2.attention.selfattn.key.weight\n",
      "bert.encoder.layer.2.attention.self.key.bias→encoder.layer.2.attention.selfattn.key.bias\n",
      "bert.encoder.layer.2.attention.self.value.weight→encoder.layer.2.attention.selfattn.value.weight\n",
      "bert.encoder.layer.2.attention.self.value.bias→encoder.layer.2.attention.selfattn.value.bias\n",
      "bert.encoder.layer.2.attention.output.dense.weight→encoder.layer.2.attention.output.dense.weight\n",
      "bert.encoder.layer.2.attention.output.dense.bias→encoder.layer.2.attention.output.dense.bias\n",
      "bert.encoder.layer.2.attention.output.LayerNorm.gamma→encoder.layer.2.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.2.attention.output.LayerNorm.beta→encoder.layer.2.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.2.intermediate.dense.weight→encoder.layer.2.intermediate.dense.weight\n",
      "bert.encoder.layer.2.intermediate.dense.bias→encoder.layer.2.intermediate.dense.bias\n",
      "bert.encoder.layer.2.output.dense.weight→encoder.layer.2.output.dense.weight\n",
      "bert.encoder.layer.2.output.dense.bias→encoder.layer.2.output.dense.bias\n",
      "bert.encoder.layer.2.output.LayerNorm.gamma→encoder.layer.2.output.LayerNorm.gamma\n",
      "bert.encoder.layer.2.output.LayerNorm.beta→encoder.layer.2.output.LayerNorm.beta\n",
      "bert.encoder.layer.3.attention.self.query.weight→encoder.layer.3.attention.selfattn.query.weight\n",
      "bert.encoder.layer.3.attention.self.query.bias→encoder.layer.3.attention.selfattn.query.bias\n",
      "bert.encoder.layer.3.attention.self.key.weight→encoder.layer.3.attention.selfattn.key.weight\n",
      "bert.encoder.layer.3.attention.self.key.bias→encoder.layer.3.attention.selfattn.key.bias\n",
      "bert.encoder.layer.3.attention.self.value.weight→encoder.layer.3.attention.selfattn.value.weight\n",
      "bert.encoder.layer.3.attention.self.value.bias→encoder.layer.3.attention.selfattn.value.bias\n",
      "bert.encoder.layer.3.attention.output.dense.weight→encoder.layer.3.attention.output.dense.weight\n",
      "bert.encoder.layer.3.attention.output.dense.bias→encoder.layer.3.attention.output.dense.bias\n",
      "bert.encoder.layer.3.attention.output.LayerNorm.gamma→encoder.layer.3.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.3.attention.output.LayerNorm.beta→encoder.layer.3.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.3.intermediate.dense.weight→encoder.layer.3.intermediate.dense.weight\n",
      "bert.encoder.layer.3.intermediate.dense.bias→encoder.layer.3.intermediate.dense.bias\n",
      "bert.encoder.layer.3.output.dense.weight→encoder.layer.3.output.dense.weight\n",
      "bert.encoder.layer.3.output.dense.bias→encoder.layer.3.output.dense.bias\n",
      "bert.encoder.layer.3.output.LayerNorm.gamma→encoder.layer.3.output.LayerNorm.gamma\n",
      "bert.encoder.layer.3.output.LayerNorm.beta→encoder.layer.3.output.LayerNorm.beta\n",
      "bert.encoder.layer.4.attention.self.query.weight→encoder.layer.4.attention.selfattn.query.weight\n",
      "bert.encoder.layer.4.attention.self.query.bias→encoder.layer.4.attention.selfattn.query.bias\n",
      "bert.encoder.layer.4.attention.self.key.weight→encoder.layer.4.attention.selfattn.key.weight\n",
      "bert.encoder.layer.4.attention.self.key.bias→encoder.layer.4.attention.selfattn.key.bias\n",
      "bert.encoder.layer.4.attention.self.value.weight→encoder.layer.4.attention.selfattn.value.weight\n",
      "bert.encoder.layer.4.attention.self.value.bias→encoder.layer.4.attention.selfattn.value.bias\n",
      "bert.encoder.layer.4.attention.output.dense.weight→encoder.layer.4.attention.output.dense.weight\n",
      "bert.encoder.layer.4.attention.output.dense.bias→encoder.layer.4.attention.output.dense.bias\n",
      "bert.encoder.layer.4.attention.output.LayerNorm.gamma→encoder.layer.4.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.4.attention.output.LayerNorm.beta→encoder.layer.4.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.4.intermediate.dense.weight→encoder.layer.4.intermediate.dense.weight\n",
      "bert.encoder.layer.4.intermediate.dense.bias→encoder.layer.4.intermediate.dense.bias\n",
      "bert.encoder.layer.4.output.dense.weight→encoder.layer.4.output.dense.weight\n",
      "bert.encoder.layer.4.output.dense.bias→encoder.layer.4.output.dense.bias\n",
      "bert.encoder.layer.4.output.LayerNorm.gamma→encoder.layer.4.output.LayerNorm.gamma\n",
      "bert.encoder.layer.4.output.LayerNorm.beta→encoder.layer.4.output.LayerNorm.beta\n",
      "bert.encoder.layer.5.attention.self.query.weight→encoder.layer.5.attention.selfattn.query.weight\n",
      "bert.encoder.layer.5.attention.self.query.bias→encoder.layer.5.attention.selfattn.query.bias\n",
      "bert.encoder.layer.5.attention.self.key.weight→encoder.layer.5.attention.selfattn.key.weight\n",
      "bert.encoder.layer.5.attention.self.key.bias→encoder.layer.5.attention.selfattn.key.bias\n",
      "bert.encoder.layer.5.attention.self.value.weight→encoder.layer.5.attention.selfattn.value.weight\n",
      "bert.encoder.layer.5.attention.self.value.bias→encoder.layer.5.attention.selfattn.value.bias\n",
      "bert.encoder.layer.5.attention.output.dense.weight→encoder.layer.5.attention.output.dense.weight\n",
      "bert.encoder.layer.5.attention.output.dense.bias→encoder.layer.5.attention.output.dense.bias\n",
      "bert.encoder.layer.5.attention.output.LayerNorm.gamma→encoder.layer.5.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.5.attention.output.LayerNorm.beta→encoder.layer.5.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.5.intermediate.dense.weight→encoder.layer.5.intermediate.dense.weight\n",
      "bert.encoder.layer.5.intermediate.dense.bias→encoder.layer.5.intermediate.dense.bias\n",
      "bert.encoder.layer.5.output.dense.weight→encoder.layer.5.output.dense.weight\n",
      "bert.encoder.layer.5.output.dense.bias→encoder.layer.5.output.dense.bias\n",
      "bert.encoder.layer.5.output.LayerNorm.gamma→encoder.layer.5.output.LayerNorm.gamma\n",
      "bert.encoder.layer.5.output.LayerNorm.beta→encoder.layer.5.output.LayerNorm.beta\n",
      "bert.encoder.layer.6.attention.self.query.weight→encoder.layer.6.attention.selfattn.query.weight\n",
      "bert.encoder.layer.6.attention.self.query.bias→encoder.layer.6.attention.selfattn.query.bias\n",
      "bert.encoder.layer.6.attention.self.key.weight→encoder.layer.6.attention.selfattn.key.weight\n",
      "bert.encoder.layer.6.attention.self.key.bias→encoder.layer.6.attention.selfattn.key.bias\n",
      "bert.encoder.layer.6.attention.self.value.weight→encoder.layer.6.attention.selfattn.value.weight\n",
      "bert.encoder.layer.6.attention.self.value.bias→encoder.layer.6.attention.selfattn.value.bias\n",
      "bert.encoder.layer.6.attention.output.dense.weight→encoder.layer.6.attention.output.dense.weight\n",
      "bert.encoder.layer.6.attention.output.dense.bias→encoder.layer.6.attention.output.dense.bias\n",
      "bert.encoder.layer.6.attention.output.LayerNorm.gamma→encoder.layer.6.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.6.attention.output.LayerNorm.beta→encoder.layer.6.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.6.intermediate.dense.weight→encoder.layer.6.intermediate.dense.weight\n",
      "bert.encoder.layer.6.intermediate.dense.bias→encoder.layer.6.intermediate.dense.bias\n",
      "bert.encoder.layer.6.output.dense.weight→encoder.layer.6.output.dense.weight\n",
      "bert.encoder.layer.6.output.dense.bias→encoder.layer.6.output.dense.bias\n",
      "bert.encoder.layer.6.output.LayerNorm.gamma→encoder.layer.6.output.LayerNorm.gamma\n",
      "bert.encoder.layer.6.output.LayerNorm.beta→encoder.layer.6.output.LayerNorm.beta\n",
      "bert.encoder.layer.7.attention.self.query.weight→encoder.layer.7.attention.selfattn.query.weight\n",
      "bert.encoder.layer.7.attention.self.query.bias→encoder.layer.7.attention.selfattn.query.bias\n",
      "bert.encoder.layer.7.attention.self.key.weight→encoder.layer.7.attention.selfattn.key.weight\n",
      "bert.encoder.layer.7.attention.self.key.bias→encoder.layer.7.attention.selfattn.key.bias\n",
      "bert.encoder.layer.7.attention.self.value.weight→encoder.layer.7.attention.selfattn.value.weight\n",
      "bert.encoder.layer.7.attention.self.value.bias→encoder.layer.7.attention.selfattn.value.bias\n",
      "bert.encoder.layer.7.attention.output.dense.weight→encoder.layer.7.attention.output.dense.weight\n",
      "bert.encoder.layer.7.attention.output.dense.bias→encoder.layer.7.attention.output.dense.bias\n",
      "bert.encoder.layer.7.attention.output.LayerNorm.gamma→encoder.layer.7.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.7.attention.output.LayerNorm.beta→encoder.layer.7.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.7.intermediate.dense.weight→encoder.layer.7.intermediate.dense.weight\n",
      "bert.encoder.layer.7.intermediate.dense.bias→encoder.layer.7.intermediate.dense.bias\n",
      "bert.encoder.layer.7.output.dense.weight→encoder.layer.7.output.dense.weight\n",
      "bert.encoder.layer.7.output.dense.bias→encoder.layer.7.output.dense.bias\n",
      "bert.encoder.layer.7.output.LayerNorm.gamma→encoder.layer.7.output.LayerNorm.gamma\n",
      "bert.encoder.layer.7.output.LayerNorm.beta→encoder.layer.7.output.LayerNorm.beta\n",
      "bert.encoder.layer.8.attention.self.query.weight→encoder.layer.8.attention.selfattn.query.weight\n",
      "bert.encoder.layer.8.attention.self.query.bias→encoder.layer.8.attention.selfattn.query.bias\n",
      "bert.encoder.layer.8.attention.self.key.weight→encoder.layer.8.attention.selfattn.key.weight\n",
      "bert.encoder.layer.8.attention.self.key.bias→encoder.layer.8.attention.selfattn.key.bias\n",
      "bert.encoder.layer.8.attention.self.value.weight→encoder.layer.8.attention.selfattn.value.weight\n",
      "bert.encoder.layer.8.attention.self.value.bias→encoder.layer.8.attention.selfattn.value.bias\n",
      "bert.encoder.layer.8.attention.output.dense.weight→encoder.layer.8.attention.output.dense.weight\n",
      "bert.encoder.layer.8.attention.output.dense.bias→encoder.layer.8.attention.output.dense.bias\n",
      "bert.encoder.layer.8.attention.output.LayerNorm.gamma→encoder.layer.8.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.8.attention.output.LayerNorm.beta→encoder.layer.8.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.8.intermediate.dense.weight→encoder.layer.8.intermediate.dense.weight\n",
      "bert.encoder.layer.8.intermediate.dense.bias→encoder.layer.8.intermediate.dense.bias\n",
      "bert.encoder.layer.8.output.dense.weight→encoder.layer.8.output.dense.weight\n",
      "bert.encoder.layer.8.output.dense.bias→encoder.layer.8.output.dense.bias\n",
      "bert.encoder.layer.8.output.LayerNorm.gamma→encoder.layer.8.output.LayerNorm.gamma\n",
      "bert.encoder.layer.8.output.LayerNorm.beta→encoder.layer.8.output.LayerNorm.beta\n",
      "bert.encoder.layer.9.attention.self.query.weight→encoder.layer.9.attention.selfattn.query.weight\n",
      "bert.encoder.layer.9.attention.self.query.bias→encoder.layer.9.attention.selfattn.query.bias\n",
      "bert.encoder.layer.9.attention.self.key.weight→encoder.layer.9.attention.selfattn.key.weight\n",
      "bert.encoder.layer.9.attention.self.key.bias→encoder.layer.9.attention.selfattn.key.bias\n",
      "bert.encoder.layer.9.attention.self.value.weight→encoder.layer.9.attention.selfattn.value.weight\n",
      "bert.encoder.layer.9.attention.self.value.bias→encoder.layer.9.attention.selfattn.value.bias\n",
      "bert.encoder.layer.9.attention.output.dense.weight→encoder.layer.9.attention.output.dense.weight\n",
      "bert.encoder.layer.9.attention.output.dense.bias→encoder.layer.9.attention.output.dense.bias\n",
      "bert.encoder.layer.9.attention.output.LayerNorm.gamma→encoder.layer.9.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.9.attention.output.LayerNorm.beta→encoder.layer.9.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.9.intermediate.dense.weight→encoder.layer.9.intermediate.dense.weight\n",
      "bert.encoder.layer.9.intermediate.dense.bias→encoder.layer.9.intermediate.dense.bias\n",
      "bert.encoder.layer.9.output.dense.weight→encoder.layer.9.output.dense.weight\n",
      "bert.encoder.layer.9.output.dense.bias→encoder.layer.9.output.dense.bias\n",
      "bert.encoder.layer.9.output.LayerNorm.gamma→encoder.layer.9.output.LayerNorm.gamma\n",
      "bert.encoder.layer.9.output.LayerNorm.beta→encoder.layer.9.output.LayerNorm.beta\n",
      "bert.encoder.layer.10.attention.self.query.weight→encoder.layer.10.attention.selfattn.query.weight\n",
      "bert.encoder.layer.10.attention.self.query.bias→encoder.layer.10.attention.selfattn.query.bias\n",
      "bert.encoder.layer.10.attention.self.key.weight→encoder.layer.10.attention.selfattn.key.weight\n",
      "bert.encoder.layer.10.attention.self.key.bias→encoder.layer.10.attention.selfattn.key.bias\n",
      "bert.encoder.layer.10.attention.self.value.weight→encoder.layer.10.attention.selfattn.value.weight\n",
      "bert.encoder.layer.10.attention.self.value.bias→encoder.layer.10.attention.selfattn.value.bias\n",
      "bert.encoder.layer.10.attention.output.dense.weight→encoder.layer.10.attention.output.dense.weight\n",
      "bert.encoder.layer.10.attention.output.dense.bias→encoder.layer.10.attention.output.dense.bias\n",
      "bert.encoder.layer.10.attention.output.LayerNorm.gamma→encoder.layer.10.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.10.attention.output.LayerNorm.beta→encoder.layer.10.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.10.intermediate.dense.weight→encoder.layer.10.intermediate.dense.weight\n",
      "bert.encoder.layer.10.intermediate.dense.bias→encoder.layer.10.intermediate.dense.bias\n",
      "bert.encoder.layer.10.output.dense.weight→encoder.layer.10.output.dense.weight\n",
      "bert.encoder.layer.10.output.dense.bias→encoder.layer.10.output.dense.bias\n",
      "bert.encoder.layer.10.output.LayerNorm.gamma→encoder.layer.10.output.LayerNorm.gamma\n",
      "bert.encoder.layer.10.output.LayerNorm.beta→encoder.layer.10.output.LayerNorm.beta\n",
      "bert.encoder.layer.11.attention.self.query.weight→encoder.layer.11.attention.selfattn.query.weight\n",
      "bert.encoder.layer.11.attention.self.query.bias→encoder.layer.11.attention.selfattn.query.bias\n",
      "bert.encoder.layer.11.attention.self.key.weight→encoder.layer.11.attention.selfattn.key.weight\n",
      "bert.encoder.layer.11.attention.self.key.bias→encoder.layer.11.attention.selfattn.key.bias\n",
      "bert.encoder.layer.11.attention.self.value.weight→encoder.layer.11.attention.selfattn.value.weight\n",
      "bert.encoder.layer.11.attention.self.value.bias→encoder.layer.11.attention.selfattn.value.bias\n",
      "bert.encoder.layer.11.attention.output.dense.weight→encoder.layer.11.attention.output.dense.weight\n",
      "bert.encoder.layer.11.attention.output.dense.bias→encoder.layer.11.attention.output.dense.bias\n",
      "bert.encoder.layer.11.attention.output.LayerNorm.gamma→encoder.layer.11.attention.output.LayerNorm.gamma\n",
      "bert.encoder.layer.11.attention.output.LayerNorm.beta→encoder.layer.11.attention.output.LayerNorm.beta\n",
      "bert.encoder.layer.11.intermediate.dense.weight→encoder.layer.11.intermediate.dense.weight\n",
      "bert.encoder.layer.11.intermediate.dense.bias→encoder.layer.11.intermediate.dense.bias\n",
      "bert.encoder.layer.11.output.dense.weight→encoder.layer.11.output.dense.weight\n",
      "bert.encoder.layer.11.output.dense.bias→encoder.layer.11.output.dense.bias\n",
      "bert.encoder.layer.11.output.LayerNorm.gamma→encoder.layer.11.output.LayerNorm.gamma\n",
      "bert.encoder.layer.11.output.LayerNorm.beta→encoder.layer.11.output.LayerNorm.beta\n",
      "bert.pooler.dense.weight→pooler.dense.weight\n",
      "bert.pooler.dense.bias→pooler.dense.bias\n"
     ]
    }
   ],
   "source": [
    "from utils.bert import get_config, BertModel, set_learned_params\n",
    "\n",
    "# モデル設定のJOSNファイルをオブジェクト変数として読み込みます\n",
    "config = get_config(file_path=\"./weights/bert_config.json\")\n",
    "\n",
    "# BERTモデルを作成します\n",
    "net_bert = BertModel(config)\n",
    "\n",
    "# BERTモデルに学習済みパラメータセットします\n",
    "net_bert = set_learned_params(\n",
    "    net_bert, weights_path=\"./weights/pytorch_model.bin\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BertForIMDb(nn.Module):\n",
    "    '''BERTモデルにIMDbのポジ・ネガを判定する部分をつなげたモデル'''\n",
    "\n",
    "    def __init__(self, net_bert):\n",
    "        super(BertForIMDb, self).__init__()\n",
    "\n",
    "        # BERTモジュール\n",
    "        self.bert = net_bert  # BERTモデル\n",
    "\n",
    "        # headにポジネガ予測を追加\n",
    "        # 入力はBERTの出力特徴量の次元、出力はポジ・ネガの2つ\n",
    "        self.cls = nn.Linear(in_features=768, out_features=2)\n",
    "\n",
    "        # 重み初期化処理\n",
    "        nn.init.normal_(self.cls.weight, std=0.02)\n",
    "        nn.init.normal_(self.cls.bias, 0)\n",
    "\n",
    "    def forward(self, input_ids, token_type_ids=None, attention_mask=None, output_all_encoded_layers=False, attention_show_flg=False):\n",
    "        '''\n",
    "        input_ids： [batch_size, sequence_length]の文章の単語IDの羅列\n",
    "        token_type_ids： [batch_size, sequence_length]の、各単語が1文目なのか、2文目なのかを示すid\n",
    "        attention_mask：Transformerのマスクと同じ働きのマスキングです\n",
    "        output_all_encoded_layers：最終出力に12段のTransformerの全部をリストで返すか、最後だけかを指定\n",
    "        attention_show_flg：Self-Attentionの重みを返すかのフラグ\n",
    "        '''\n",
    "\n",
    "        # BERTの基本モデル部分の順伝搬\n",
    "        # 順伝搬させる\n",
    "        if attention_show_flg == True:\n",
    "            '''attention_showのときは、attention_probsもリターンする'''\n",
    "            encoded_layers, pooled_output, attention_probs = self.bert(\n",
    "                input_ids, token_type_ids, attention_mask, output_all_encoded_layers, attention_show_flg)\n",
    "        elif attention_show_flg == False:\n",
    "            encoded_layers, pooled_output = self.bert(\n",
    "                input_ids, token_type_ids, attention_mask, output_all_encoded_layers, attention_show_flg)\n",
    "\n",
    "        # 入力文章の1単語目[CLS]の特徴量を使用して、ポジ・ネガを分類します\n",
    "        vec_0 = encoded_layers[:, 0, :]\n",
    "        vec_0 = vec_0.view(-1, 768)  # sizeを[batch_size, hidden_sizeに変換\n",
    "        out = self.cls(vec_0)\n",
    "\n",
    "        # attention_showのときは、attention_probs（1番最後の）もリターンする\n",
    "        if attention_show_flg == True:\n",
    "            return out, attention_probs\n",
    "        elif attention_show_flg == False:\n",
    "            return out\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ネットワーク設定完了\n"
     ]
    }
   ],
   "source": [
    "# モデル構築\n",
    "net = BertForIMDb(net_bert)\n",
    "\n",
    "# 訓練モードに設定\n",
    "net.train()\n",
    "\n",
    "print('ネットワーク設定完了')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERTのファインチューニングに向けた設定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 勾配計算を最後のBertLayerモジュールと追加した分類アダプターのみ実行\n",
    "\n",
    "# 1. まず全部を、勾配計算Falseにしてしまう\n",
    "for name, param in net.named_parameters():\n",
    "    param.requires_grad = False\n",
    "\n",
    "# 2. 最後のBertLayerモジュールを勾配計算ありに変更\n",
    "for name, param in net.bert.encoder.layer[-1].named_parameters():\n",
    "    param.requires_grad = True\n",
    "\n",
    "# 3. 識別器を勾配計算ありに変更\n",
    "for name, param in net.cls.named_parameters():\n",
    "    param.requires_grad = True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 最適化手法の設定\n",
    "\n",
    "# BERTの元の部分はファインチューニング\n",
    "optimizer = optim.Adam([\n",
    "    {'params': net.bert.encoder.layer[-1].parameters(), 'lr': 5e-5},\n",
    "    {'params': net.cls.parameters(), 'lr': 5e-5}\n",
    "], betas=(0.9, 0.999))\n",
    "\n",
    "# 損失関数の設定\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "# nn.LogSoftmax()を計算してからnn.NLLLoss(negative log likelihood loss)を計算\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学習・検証を実施"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# モデルを学習させる関数を作成\n",
    "\n",
    "\n",
    "def train_model(net, dataloaders_dict, criterion, optimizer, num_epochs):\n",
    "\n",
    "    # GPUが使えるかを確認\n",
    "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "    print(\"使用デバイス：\", device)\n",
    "    print('-----start-------')\n",
    "\n",
    "    # ネットワークをGPUへ\n",
    "    net.to(device)\n",
    "\n",
    "    # ネットワークがある程度固定であれば、高速化させる\n",
    "    torch.backends.cudnn.benchmark = True\n",
    "\n",
    "    # ミニバッチのサイズ\n",
    "    batch_size = dataloaders_dict[\"train\"].batch_size\n",
    "\n",
    "    # epochのループ\n",
    "    for epoch in range(num_epochs):\n",
    "        # epochごとの訓練と検証のループ\n",
    "        for phase in ['train', 'val']:\n",
    "            if phase == 'train':\n",
    "                net.train()  # モデルを訓練モードに\n",
    "            else:\n",
    "                net.eval()   # モデルを検証モードに\n",
    "\n",
    "            epoch_loss = 0.0  # epochの損失和\n",
    "            epoch_corrects = 0  # epochの正解数\n",
    "            iteration = 1\n",
    "\n",
    "            # 開始時刻を保存\n",
    "            t_epoch_start = time.time()\n",
    "            t_iter_start = time.time()\n",
    "\n",
    "            # データローダーからミニバッチを取り出すループ\n",
    "            for batch in (dataloaders_dict[phase]):\n",
    "                # batchはTextとLableの辞書型変数\n",
    "\n",
    "                # GPUが使えるならGPUにデータを送る\n",
    "                inputs = batch.Text[0].to(device)  # 文章\n",
    "                labels = batch.Label.to(device)  # ラベル\n",
    "\n",
    "                # optimizerを初期化\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                # 順伝搬（forward）計算\n",
    "                with torch.set_grad_enabled(phase == 'train'):\n",
    "\n",
    "                    # BertForIMDbに入力\n",
    "                    outputs = net(inputs, token_type_ids=None, attention_mask=None,\n",
    "                                  output_all_encoded_layers=False, attention_show_flg=False)\n",
    "\n",
    "                    loss = criterion(outputs, labels)  # 損失を計算\n",
    "\n",
    "                    _, preds = torch.max(outputs, 1)  # ラベルを予測\n",
    "\n",
    "                    # 訓練時はバックプロパゲーション\n",
    "                    if phase == 'train':\n",
    "                        loss.backward()\n",
    "                        optimizer.step()\n",
    "\n",
    "                        if (iteration % 10 == 0):  # 10iterに1度、lossを表示\n",
    "                            t_iter_finish = time.time()\n",
    "                            duration = t_iter_finish - t_iter_start\n",
    "                            acc = (torch.sum(preds == labels.data)\n",
    "                                   ).double()/batch_size\n",
    "                            print('イテレーション {} || Loss: {:.4f} || 10iter: {:.4f} sec. || 本イテレーションの正解率：{}'.format(\n",
    "                                iteration, loss.item(), duration, acc))\n",
    "                            t_iter_start = time.time()\n",
    "\n",
    "                    iteration += 1\n",
    "\n",
    "                    # 損失と正解数の合計を更新\n",
    "                    epoch_loss += loss.item() * batch_size\n",
    "                    epoch_corrects += torch.sum(preds == labels.data)\n",
    "\n",
    "            # epochごとのlossと正解率\n",
    "            t_epoch_finish = time.time()\n",
    "            epoch_loss = epoch_loss / len(dataloaders_dict[phase].dataset)\n",
    "            epoch_acc = epoch_corrects.double(\n",
    "            ) / len(dataloaders_dict[phase].dataset)\n",
    "\n",
    "            print('Epoch {}/{} | {:^5} |  Loss: {:.4f} Acc: {:.4f}'.format(epoch+1, num_epochs,\n",
    "                                                                           phase, epoch_loss, epoch_acc))\n",
    "            t_epoch_start = time.time()\n",
    "\n",
    "    return net\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "使用デバイス： cuda:0\n",
      "-----start-------\n",
      "イテレーション 10 || Loss: 0.7256 || 10iter: 11.7782 sec. || 本イテレーションの正解率：0.53125\n",
      "イテレーション 20 || Loss: 0.6693 || 10iter: 11.0131 sec. || 本イテレーションの正解率：0.59375\n",
      "イテレーション 30 || Loss: 0.6792 || 10iter: 11.0456 sec. || 本イテレーションの正解率：0.5625\n",
      "イテレーション 40 || Loss: 0.6383 || 10iter: 11.1244 sec. || 本イテレーションの正解率：0.625\n",
      "イテレーション 50 || Loss: 0.6787 || 10iter: 11.0196 sec. || 本イテレーションの正解率：0.5625\n",
      "イテレーション 60 || Loss: 0.6298 || 10iter: 11.1094 sec. || 本イテレーションの正解率：0.625\n",
      "イテレーション 70 || Loss: 0.5353 || 10iter: 11.1385 sec. || 本イテレーションの正解率：0.6875\n",
      "イテレーション 80 || Loss: 0.6848 || 10iter: 11.1681 sec. || 本イテレーションの正解率：0.6875\n",
      "イテレーション 90 || Loss: 0.4199 || 10iter: 11.0776 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 100 || Loss: 0.3817 || 10iter: 11.1341 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 110 || Loss: 0.4331 || 10iter: 11.1551 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 120 || Loss: 0.5997 || 10iter: 11.1967 sec. || 本イテレーションの正解率：0.65625\n",
      "イテレーション 130 || Loss: 0.3853 || 10iter: 11.1050 sec. || 本イテレーションの正解率：0.75\n",
      "イテレーション 140 || Loss: 0.2892 || 10iter: 11.1738 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 150 || Loss: 0.2563 || 10iter: 11.1655 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 160 || Loss: 0.3050 || 10iter: 11.2534 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 170 || Loss: 0.3606 || 10iter: 11.1159 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 180 || Loss: 0.3301 || 10iter: 11.1913 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 190 || Loss: 0.5569 || 10iter: 11.1774 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 200 || Loss: 0.3452 || 10iter: 11.2277 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 210 || Loss: 0.3165 || 10iter: 11.1246 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 220 || Loss: 0.3182 || 10iter: 11.1835 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 230 || Loss: 0.1521 || 10iter: 11.2006 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 240 || Loss: 0.3232 || 10iter: 11.2354 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 250 || Loss: 0.3430 || 10iter: 11.1352 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 260 || Loss: 0.4065 || 10iter: 11.1907 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 270 || Loss: 0.2681 || 10iter: 11.2049 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 280 || Loss: 0.1898 || 10iter: 11.2581 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 290 || Loss: 0.2194 || 10iter: 11.1388 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 300 || Loss: 0.1511 || 10iter: 11.1931 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 310 || Loss: 0.3068 || 10iter: 11.2035 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 320 || Loss: 0.3755 || 10iter: 11.2729 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 330 || Loss: 0.1526 || 10iter: 11.1273 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 340 || Loss: 0.3795 || 10iter: 11.2047 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 350 || Loss: 0.2085 || 10iter: 11.1874 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 360 || Loss: 0.2135 || 10iter: 11.2545 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 370 || Loss: 0.2183 || 10iter: 11.1362 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 380 || Loss: 0.2215 || 10iter: 11.2079 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 390 || Loss: 0.5849 || 10iter: 11.1912 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 400 || Loss: 0.1269 || 10iter: 11.2646 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 410 || Loss: 0.1615 || 10iter: 11.1455 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 420 || Loss: 0.3387 || 10iter: 11.1992 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 430 || Loss: 0.2105 || 10iter: 11.1931 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 440 || Loss: 0.3881 || 10iter: 11.2665 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 450 || Loss: 0.2593 || 10iter: 11.1327 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 460 || Loss: 0.4442 || 10iter: 11.1944 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 470 || Loss: 0.2436 || 10iter: 11.1977 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 480 || Loss: 0.2993 || 10iter: 11.2529 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 490 || Loss: 0.2294 || 10iter: 11.1458 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 500 || Loss: 0.2259 || 10iter: 11.1952 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 510 || Loss: 0.1810 || 10iter: 11.1994 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 520 || Loss: 0.2542 || 10iter: 11.2446 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 530 || Loss: 0.2131 || 10iter: 11.1294 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 540 || Loss: 0.1907 || 10iter: 11.1872 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 550 || Loss: 0.3461 || 10iter: 11.1872 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 560 || Loss: 0.1238 || 10iter: 11.2553 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 570 || Loss: 0.2882 || 10iter: 11.1584 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 580 || Loss: 0.3873 || 10iter: 11.2002 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 590 || Loss: 0.3681 || 10iter: 11.1936 sec. || 本イテレーションの正解率：0.75\n",
      "イテレーション 600 || Loss: 0.1959 || 10iter: 11.2508 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 610 || Loss: 0.2095 || 10iter: 11.1414 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 620 || Loss: 0.2241 || 10iter: 11.1957 sec. || 本イテレーションの正解率：0.90625\n",
      "Epoch 1/2 | train |  Loss: 0.3441 Acc: 0.8409\n",
      "Epoch 1/2 |  val  |  Loss: 0.2963 Acc: 0.8856\n",
      "イテレーション 10 || Loss: 0.2805 || 10iter: 11.2160 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 20 || Loss: 0.2351 || 10iter: 11.2093 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 30 || Loss: 0.1772 || 10iter: 11.2420 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 40 || Loss: 0.1911 || 10iter: 11.1512 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 50 || Loss: 0.1684 || 10iter: 11.1919 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 60 || Loss: 0.1275 || 10iter: 11.2048 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 70 || Loss: 0.2191 || 10iter: 11.2582 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 80 || Loss: 0.3155 || 10iter: 11.1232 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 90 || Loss: 0.3370 || 10iter: 11.1908 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 100 || Loss: 0.1253 || 10iter: 11.1905 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 110 || Loss: 0.2714 || 10iter: 11.2712 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 120 || Loss: 0.3624 || 10iter: 11.1373 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 130 || Loss: 0.4109 || 10iter: 11.2082 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 140 || Loss: 0.1307 || 10iter: 11.1969 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 150 || Loss: 0.2239 || 10iter: 11.2546 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 160 || Loss: 0.4943 || 10iter: 11.1289 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 170 || Loss: 0.2647 || 10iter: 11.1941 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 180 || Loss: 0.2296 || 10iter: 11.1951 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 190 || Loss: 0.3329 || 10iter: 11.2633 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 200 || Loss: 0.3621 || 10iter: 11.1550 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 210 || Loss: 0.2394 || 10iter: 11.2055 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 220 || Loss: 0.2060 || 10iter: 11.2001 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 230 || Loss: 0.3101 || 10iter: 11.2621 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 240 || Loss: 0.3040 || 10iter: 11.1474 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 250 || Loss: 0.4133 || 10iter: 11.1965 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 260 || Loss: 0.1730 || 10iter: 11.1981 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 270 || Loss: 0.2649 || 10iter: 11.2427 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 280 || Loss: 0.2397 || 10iter: 11.1409 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 290 || Loss: 0.2651 || 10iter: 11.2034 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 300 || Loss: 0.2927 || 10iter: 11.2037 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 310 || Loss: 0.2181 || 10iter: 11.2452 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 320 || Loss: 0.3769 || 10iter: 11.1387 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 330 || Loss: 0.1898 || 10iter: 11.1909 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 340 || Loss: 0.2395 || 10iter: 11.2165 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 350 || Loss: 0.3071 || 10iter: 11.2613 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 360 || Loss: 0.2082 || 10iter: 11.1440 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 370 || Loss: 0.2767 || 10iter: 11.2011 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 380 || Loss: 0.2325 || 10iter: 11.2013 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 390 || Loss: 0.2175 || 10iter: 11.2729 sec. || 本イテレーションの正解率：0.9375\n",
      "イテレーション 400 || Loss: 0.3782 || 10iter: 11.1415 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 410 || Loss: 0.4511 || 10iter: 11.2136 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 420 || Loss: 0.3259 || 10iter: 11.1942 sec. || 本イテレーションの正解率：0.78125\n",
      "イテレーション 430 || Loss: 0.1894 || 10iter: 11.2594 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 440 || Loss: 0.0983 || 10iter: 11.1480 sec. || 本イテレーションの正解率：1.0\n",
      "イテレーション 450 || Loss: 0.2988 || 10iter: 11.2043 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 460 || Loss: 0.1123 || 10iter: 11.2004 sec. || 本イテレーションの正解率：0.9375\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "イテレーション 470 || Loss: 0.2237 || 10iter: 11.2497 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 480 || Loss: 0.3752 || 10iter: 11.1577 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 490 || Loss: 0.3263 || 10iter: 11.1995 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 500 || Loss: 0.2696 || 10iter: 11.2002 sec. || 本イテレーションの正解率：0.90625\n",
      "イテレーション 510 || Loss: 0.1998 || 10iter: 11.2428 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 520 || Loss: 0.3297 || 10iter: 11.1420 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 530 || Loss: 0.3994 || 10iter: 11.1790 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 540 || Loss: 0.0992 || 10iter: 11.1910 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 550 || Loss: 0.4392 || 10iter: 11.2618 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 560 || Loss: 0.1908 || 10iter: 11.1351 sec. || 本イテレーションの正解率：0.96875\n",
      "イテレーション 570 || Loss: 0.3511 || 10iter: 11.1987 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 580 || Loss: 0.2716 || 10iter: 11.2105 sec. || 本イテレーションの正解率：0.875\n",
      "イテレーション 590 || Loss: 0.3917 || 10iter: 11.2555 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 600 || Loss: 0.3501 || 10iter: 11.1452 sec. || 本イテレーションの正解率：0.8125\n",
      "イテレーション 610 || Loss: 0.2081 || 10iter: 11.1994 sec. || 本イテレーションの正解率：0.84375\n",
      "イテレーション 620 || Loss: 0.1388 || 10iter: 11.2088 sec. || 本イテレーションの正解率：0.90625\n",
      "Epoch 2/2 | train |  Loss: 0.2566 Acc: 0.8957\n",
      "Epoch 2/2 |  val  |  Loss: 0.2589 Acc: 0.9004\n"
     ]
    }
   ],
   "source": [
    "# 学習・検証を実行する。1epochに20分ほどかかります\n",
    "num_epochs = 2\n",
    "net_trained = train_model(net, dataloaders_dict,\n",
    "                          criterion, optimizer, num_epochs=num_epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 学習したネットワークパラメータを保存します\n",
    "save_path = './weights/bert_fine_tuning_IMDb.pth'\n",
    "torch.save(net_trained.state_dict(), save_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [13:41<00:00,  1.05s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "テストデータ25000個での正解率：0.9038\n"
     ]
    }
   ],
   "source": [
    "# テストデータでの正解率を求める\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "net_trained.eval()   # モデルを検証モードに\n",
    "net_trained.to(device)  # GPUが使えるならGPUへ送る\n",
    "\n",
    "# epochの正解数を記録する変数\n",
    "epoch_corrects = 0\n",
    "\n",
    "for batch in tqdm(test_dl):  # testデータのDataLoader\n",
    "    # batchはTextとLableの辞書オブジェクト\n",
    "    # GPUが使えるならGPUにデータを送る\n",
    "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "    inputs = batch.Text[0].to(device)  # 文章\n",
    "    labels = batch.Label.to(device)  # ラベル\n",
    "\n",
    "    # 順伝搬（forward）計算\n",
    "    with torch.set_grad_enabled(False):\n",
    "\n",
    "        # BertForIMDbに入力\n",
    "        outputs = net_trained(inputs, token_type_ids=None, attention_mask=None,\n",
    "                              output_all_encoded_layers=False, attention_show_flg=False)\n",
    "\n",
    "        loss = criterion(outputs, labels)  # 損失を計算\n",
    "        _, preds = torch.max(outputs, 1)  # ラベルを予測\n",
    "        epoch_corrects += torch.sum(preds == labels.data)  # 正解数の合計を更新\n",
    "\n",
    "# 正解率\n",
    "epoch_acc = epoch_corrects.double() / len(test_dl.dataset)\n",
    "\n",
    "print('テストデータ{}個での正解率：{:.4f}'.format(len(test_dl.dataset), epoch_acc))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attentionの可視化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# batch_sizeを64にしたテストデータでDataLoaderを作成\n",
    "batch_size = 64\n",
    "test_dl = torchtext.data.Iterator(\n",
    "    test_ds, batch_size=batch_size, train=False, sort=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BertForIMDbで処理\n",
    "\n",
    "# ミニバッチの用意\n",
    "batch = next(iter(test_dl))\n",
    "\n",
    "# GPUが使えるならGPUにデータを送る\n",
    "inputs = batch.Text[0].to(device)  # 文章\n",
    "labels = batch.Label.to(device)  # ラベル\n",
    "\n",
    "outputs, attention_probs = net_trained(inputs, token_type_ids=None, attention_mask=None,\n",
    "                                       output_all_encoded_layers=False, attention_show_flg=True)\n",
    "\n",
    "_, preds = torch.max(outputs, 1)  # ラベルを予測\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# HTMLを作成する関数を実装\n",
    "\n",
    "\n",
    "def highlight(word, attn):\n",
    "    \"Attentionの値が大きいと文字の背景が濃い赤になるhtmlを出力させる関数\"\n",
    "\n",
    "    html_color = '#%02X%02X%02X' % (\n",
    "        255, int(255*(1 - attn)), int(255*(1 - attn)))\n",
    "    return '<span style=\"background-color: {}\"> {}</span>'.format(html_color, word)\n",
    "\n",
    "\n",
    "def mk_html(index, batch, preds, normlized_weights, TEXT):\n",
    "    \"HTMLデータを作成する\"\n",
    "\n",
    "    # indexの結果を抽出\n",
    "    sentence = batch.Text[0][index]  # 文章\n",
    "    label = batch.Label[index]  # ラベル\n",
    "    pred = preds[index]  # 予測\n",
    "\n",
    "    # ラベルと予測結果を文字に置き換え\n",
    "    if label == 0:\n",
    "        label_str = \"Negative\"\n",
    "    else:\n",
    "        label_str = \"Positive\"\n",
    "\n",
    "    if pred == 0:\n",
    "        pred_str = \"Negative\"\n",
    "    else:\n",
    "        pred_str = \"Positive\"\n",
    "\n",
    "    # 表示用のHTMLを作成する\n",
    "    html = '正解ラベル：{}<br>推論ラベル：{}<br><br>'.format(label_str, pred_str)\n",
    "\n",
    "    # Self-Attentionの重みを可視化。Multi-Headが12個なので、12種類のアテンションが存在\n",
    "    for i in range(12):\n",
    "\n",
    "        # indexのAttentionを抽出と規格化\n",
    "        # 0単語目[CLS]の、i番目のMulti-Head Attentionを取り出す\n",
    "        # indexはミニバッチの何個目のデータかをしめす\n",
    "        attens = normlized_weights[index, i, 0, :]\n",
    "        attens /= attens.max()\n",
    "\n",
    "        html += '[BERTのAttentionを可視化_' + str(i+1) + ']<br>'\n",
    "        for word, attn in zip(sentence, attens):\n",
    "\n",
    "            # 単語が[SEP]の場合は文章が終わりなのでbreak\n",
    "            if tokenizer_bert.convert_ids_to_tokens([word.numpy().tolist()])[0] == \"[SEP]\":\n",
    "                break\n",
    "\n",
    "            # 関数highlightで色をつける、関数tokenizer_bert.convert_ids_to_tokensでIDを単語に戻す\n",
    "            html += highlight(tokenizer_bert.convert_ids_to_tokens(\n",
    "                [word.numpy().tolist()])[0], attn)\n",
    "        html += \"<br><br>\"\n",
    "\n",
    "    # 12種類のAttentionの平均を求める。最大値で規格化\n",
    "    all_attens = attens*0  # all_attensという変数を作成する\n",
    "    for i in range(12):\n",
    "        attens += normlized_weights[index, i, 0, :]\n",
    "    attens /= attens.max()\n",
    "\n",
    "    html += '[BERTのAttentionを可視化_ALL]<br>'\n",
    "    for word, attn in zip(sentence, attens):\n",
    "\n",
    "        # 単語が[SEP]の場合は文章が終わりなのでbreak\n",
    "        if tokenizer_bert.convert_ids_to_tokens([word.numpy().tolist()])[0] == \"[SEP]\":\n",
    "            break\n",
    "\n",
    "        # 関数highlightで色をつける、関数tokenizer_bert.convert_ids_to_tokensでIDを単語に戻す\n",
    "        html += highlight(tokenizer_bert.convert_ids_to_tokens(\n",
    "            [word.numpy().tolist()])[0], attn)\n",
    "    html += \"<br><br>\"\n",
    "\n",
    "    return html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "正解ラベル：Positive<br>推論ラベル：Positive<br><br>[BERTのAttentionを可視化_1]<br><span style=\"background-color: #FFF2F2\"> [CLS]</span><span style=\"background-color: #FFA9A9\"> a</span><span style=\"background-color: #FFE8E8\"> wonderful</span><span style=\"background-color: #FFF6F6\"> film</span><span style=\"background-color: #FFA7A7\"> a</span><span style=\"background-color: #FFE7E7\"> charming</span><span style=\"background-color: #FFF1F1\"> ,</span><span style=\"background-color: #FFF5F5\"> qui</span><span style=\"background-color: #FFF6F6\"> ##rky</span><span style=\"background-color: #FFF8F8\"> family</span><span style=\"background-color: #FFF3F3\"> story</span><span style=\"background-color: #FFCACA\"> .</span><span style=\"background-color: #FF9E9E\"> a</span><span style=\"background-color: #FFD8D8\"> cross</span><span style=\"background-color: #FFFAFA\"> country</span><span style=\"background-color: #FFD7D7\"> journey</span><span style=\"background-color: #FFDCDC\"> filled</span><span style=\"background-color: #FFCECE\"> with</span><span style=\"background-color: #FFEAEA\"> lots</span><span style=\"background-color: #FFD4D4\"> of</span><span style=\"background-color: #FFD2D2\"> interesting</span><span style=\"background-color: #FFBCBC\"> ,</span><span style=\"background-color: #FF9C9C\"> odd</span><span style=\"background-color: #FFC3C3\"> ##ball</span><span style=\"background-color: #FF0000\"> stops</span><span style=\"background-color: #FF7575\"> along</span><span style=\"background-color: #FFEAEA\"> the</span><span style=\"background-color: #FFE1E1\"> way</span><span style=\"background-color: #FFABAB\"> several</span><span style=\"background-color: #FFF3F3\"> very</span><span style=\"background-color: #FF3A3A\"> cool</span><span style=\"background-color: #FFD9D9\"> cameo</span><span style=\"background-color: #FFDDDD\"> ##s</span><span style=\"background-color: #FF6969\"> .</span><span style=\"background-color: #FFE4E4\"> great</span><span style=\"background-color: #FFA2A2\"> cast</span><span style=\"background-color: #FFAEAE\"> led</span><span style=\"background-color: #FFEBEB\"> by</span><span style=\"background-color: #FFFAFA\"> rod</span><span style=\"background-color: #FFFDFD\"> ste</span><span style=\"background-color: #FFFCFC\"> ##iger</span><span style=\"background-color: #FF9292\"> carries</span><span style=\"background-color: #FFF8F8\"> the</span><span style=\"background-color: #FFE1E1\"> film</span><span style=\"background-color: #FF7F7F\"> along</span><span style=\"background-color: #FFDDDD\"> and</span><span style=\"background-color: #FF7B7B\"> leads</span><span style=\"background-color: #FFB7B7\"> to</span><span style=\"background-color: #FFB3B3\"> a</span><span style=\"background-color: #FFE2E2\"> surprise</span><span style=\"background-color: #FF6F6F\"> ending</span><span style=\"background-color: #FF4E4E\"> .</span><span style=\"background-color: #FFEBEB\"> well</span><span style=\"background-color: #FF8D8D\"> directed</span><span style=\"background-color: #FFE4E4\"> shot</span><span style=\"background-color: #FFEFEF\"> a</span><span style=\"background-color: #FFFCFC\"> really</span><span style=\"background-color: #FFEDED\"> nice</span><span style=\"background-color: #FFFAFA\"> movie</span><span style=\"background-color: #FF5959\"> .</span><br><br>[BERTのAttentionを可視化_2]<br><span style=\"background-color: #FFEFEF\"> [CLS]</span><span style=\"background-color: #FFF9F9\"> a</span><span style=\"background-color: #FFF9F9\"> wonderful</span><span style=\"background-color: #FF9292\"> film</span><span style=\"background-color: #FFFAFA\"> a</span><span style=\"background-color: #FFFBFB\"> charming</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> qui</span><span style=\"background-color: #FFFCFC\"> ##rky</span><span style=\"background-color: #FFF7F7\"> family</span><span style=\"background-color: #FFF4F4\"> story</span><span style=\"background-color: #FFFCFC\"> .</span><span style=\"background-color: #FFF9F9\"> a</span><span style=\"background-color: #FFF1F1\"> cross</span><span style=\"background-color: #FFEEEE\"> country</span><span style=\"background-color: #FFE0E0\"> journey</span><span style=\"background-color: #FFFDFD\"> filled</span><span style=\"background-color: #FFFDFD\"> with</span><span style=\"background-color: #FFFEFE\"> lots</span><span style=\"background-color: #FFFDFD\"> of</span><span style=\"background-color: #FFFBFB\"> interesting</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFFCFC\"> odd</span><span style=\"background-color: #FFFAFA\"> ##ball</span><span style=\"background-color: #FFF9F9\"> stops</span><span style=\"background-color: #FFFDFD\"> along</span><span style=\"background-color: #FFF3F3\"> the</span><span style=\"background-color: #FFEBEB\"> way</span><span style=\"background-color: #FFFEFE\"> several</span><span style=\"background-color: #FFFDFD\"> very</span><span style=\"background-color: #FFF9F9\"> cool</span><span style=\"background-color: #FFFBFB\"> cameo</span><span style=\"background-color: #FFFDFD\"> ##s</span><span style=\"background-color: #FFDEDE\"> .</span><span style=\"background-color: #FFF7F7\"> great</span><span style=\"background-color: #FF5858\"> cast</span><span style=\"background-color: #FFF9F9\"> led</span><span style=\"background-color: #FFE4E4\"> by</span><span style=\"background-color: #FFF5F5\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFDFD\"> ##iger</span><span style=\"background-color: #FFFBFB\"> carries</span><span style=\"background-color: #FFBFBF\"> the</span><span style=\"background-color: #FF0000\"> film</span><span style=\"background-color: #FFF7F7\"> along</span><span style=\"background-color: #FFFAFA\"> and</span><span style=\"background-color: #FFFDFD\"> leads</span><span style=\"background-color: #FFFAFA\"> to</span><span style=\"background-color: #FFF3F3\"> a</span><span style=\"background-color: #FFFAFA\"> surprise</span><span style=\"background-color: #FFD5D5\"> ending</span><span style=\"background-color: #FFD8D8\"> .</span><span style=\"background-color: #FFFAFA\"> well</span><span style=\"background-color: #FFF8F8\"> directed</span><span style=\"background-color: #FFF2F2\"> shot</span><span style=\"background-color: #FFF2F2\"> a</span><span style=\"background-color: #FFFDFD\"> really</span><span style=\"background-color: #FFF2F2\"> nice</span><span style=\"background-color: #FF3131\"> movie</span><span style=\"background-color: #FFD9D9\"> .</span><br><br>[BERTのAttentionを可視化_3]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FF0000\"> a</span><span style=\"background-color: #FFF4F4\"> wonderful</span><span style=\"background-color: #FFF1F1\"> film</span><span style=\"background-color: #FF3939\"> a</span><span style=\"background-color: #FFF3F3\"> charming</span><span style=\"background-color: #FF9E9E\"> ,</span><span style=\"background-color: #FFFCFC\"> qui</span><span style=\"background-color: #FFF3F3\"> ##rky</span><span style=\"background-color: #FFFEFE\"> family</span><span style=\"background-color: #FFECEC\"> story</span><span style=\"background-color: #FFEAEA\"> .</span><span style=\"background-color: #FF4545\"> a</span><span style=\"background-color: #FFFCFC\"> cross</span><span style=\"background-color: #FFFDFD\"> country</span><span style=\"background-color: #FFF9F9\"> journey</span><span style=\"background-color: #FFE7E7\"> filled</span><span style=\"background-color: #FFD9D9\"> with</span><span style=\"background-color: #FFFAFA\"> lots</span><span style=\"background-color: #FFEDED\"> of</span><span style=\"background-color: #FFE9E9\"> interesting</span><span style=\"background-color: #FFA6A6\"> ,</span><span style=\"background-color: #FFFAFA\"> odd</span><span style=\"background-color: #FFE8E8\"> ##ball</span><span style=\"background-color: #FFF9F9\"> stops</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFF7F7\"> the</span><span style=\"background-color: #FFFEFE\"> way</span><span style=\"background-color: #FFFCFC\"> several</span><span style=\"background-color: #FFE5E5\"> very</span><span style=\"background-color: #FFF6F6\"> cool</span><span style=\"background-color: #FFF8F8\"> cameo</span><span style=\"background-color: #FFF8F8\"> ##s</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFDFD\"> great</span><span style=\"background-color: #FFF5F5\"> cast</span><span style=\"background-color: #FFF9F9\"> led</span><span style=\"background-color: #FFF9F9\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFFEFE\"> carries</span><span style=\"background-color: #FFC3C3\"> the</span><span style=\"background-color: #FFF9F9\"> film</span><span style=\"background-color: #FFFBFB\"> along</span><span style=\"background-color: #FFB9B9\"> and</span><span style=\"background-color: #FFFDFD\"> leads</span><span style=\"background-color: #FFF1F1\"> to</span><span style=\"background-color: #FF8A8A\"> a</span><span style=\"background-color: #FFF4F4\"> surprise</span><span style=\"background-color: #FFEEEE\"> ending</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE7E7\"> well</span><span style=\"background-color: #FFFBFB\"> directed</span><span style=\"background-color: #FFE0E0\"> shot</span><span style=\"background-color: #FF3C3C\"> a</span><span style=\"background-color: #FFECEC\"> really</span><span style=\"background-color: #FFEDED\"> nice</span><span style=\"background-color: #FFF3F3\"> movie</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_4]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FF0000\"> a</span><span style=\"background-color: #FFD4D4\"> wonderful</span><span style=\"background-color: #FFA5A5\"> film</span><span style=\"background-color: #FFD8D8\"> a</span><span style=\"background-color: #FFE3E3\"> charming</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFFBFB\"> qui</span><span style=\"background-color: #FFF8F8\"> ##rky</span><span style=\"background-color: #FFF6F6\"> family</span><span style=\"background-color: #FFEDED\"> story</span><span style=\"background-color: #FFF8F8\"> .</span><span style=\"background-color: #FFD8D8\"> a</span><span style=\"background-color: #FFFEFE\"> cross</span><span style=\"background-color: #FFFDFD\"> country</span><span style=\"background-color: #FFFAFA\"> journey</span><span style=\"background-color: #FFFCFC\"> filled</span><span style=\"background-color: #FFFDFD\"> with</span><span style=\"background-color: #FFFCFC\"> lots</span><span style=\"background-color: #FFFCFC\"> of</span><span style=\"background-color: #FFF6F6\"> interesting</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFDFD\"> odd</span><span style=\"background-color: #FFFCFC\"> ##ball</span><span style=\"background-color: #FFF8F8\"> stops</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFFEFE\"> way</span><span style=\"background-color: #FFFDFD\"> several</span><span style=\"background-color: #FFF9F9\"> very</span><span style=\"background-color: #FFFBFB\"> cool</span><span style=\"background-color: #FFF4F4\"> cameo</span><span style=\"background-color: #FFFAFA\"> ##s</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFDFD\"> great</span><span style=\"background-color: #FFFCFC\"> cast</span><span style=\"background-color: #FFFEFE\"> led</span><span style=\"background-color: #FFFEFE\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFFEFE\"> carries</span><span style=\"background-color: #FFE0E0\"> the</span><span style=\"background-color: #FFDFDF\"> film</span><span style=\"background-color: #FFFCFC\"> along</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFEFE\"> leads</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFF7F7\"> a</span><span style=\"background-color: #FFFDFD\"> surprise</span><span style=\"background-color: #FFF7F7\"> ending</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE3E3\"> well</span><span style=\"background-color: #FFFBFB\"> directed</span><span style=\"background-color: #FFEFEF\"> shot</span><span style=\"background-color: #FFCACA\"> a</span><span style=\"background-color: #FFFDFD\"> really</span><span style=\"background-color: #FFF5F5\"> nice</span><span style=\"background-color: #FFDADA\"> movie</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_5]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFE8E8\"> a</span><span style=\"background-color: #FFA8A8\"> wonderful</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFBFB\"> a</span><span style=\"background-color: #FFF8F8\"> charming</span><span style=\"background-color: #FFE8E8\"> ,</span><span style=\"background-color: #FFFEFE\"> qui</span><span style=\"background-color: #FFFEFE\"> ##rky</span><span style=\"background-color: #FFFCFC\"> family</span><span style=\"background-color: #FFFEFE\"> story</span><span style=\"background-color: #FFE8E8\"> .</span><span style=\"background-color: #FFFDFD\"> a</span><span style=\"background-color: #FFFEFE\"> cross</span><span style=\"background-color: #FFFEFE\"> country</span><span style=\"background-color: #FFFDFD\"> journey</span><span style=\"background-color: #FFFEFE\"> filled</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFAFA\"> lots</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFF7F7\"> ,</span><span style=\"background-color: #FFFEFE\"> odd</span><span style=\"background-color: #FFFEFE\"> ##ball</span><span style=\"background-color: #FFFEFE\"> stops</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> way</span><span style=\"background-color: #FFFEFE\"> several</span><span style=\"background-color: #FFEEEE\"> very</span><span style=\"background-color: #FFFDFD\"> cool</span><span style=\"background-color: #FFFEFE\"> cameo</span><span style=\"background-color: #FFFEFE\"> ##s</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFECEC\"> great</span><span style=\"background-color: #FFFDFD\"> cast</span><span style=\"background-color: #FFFEFE\"> led</span><span style=\"background-color: #FFFEFE\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFFEFE\"> carries</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFEFE\"> leads</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFDFD\"> a</span><span style=\"background-color: #FFF7F7\"> surprise</span><span style=\"background-color: #FFFEFE\"> ending</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FF3939\"> well</span><span style=\"background-color: #FFFDFD\"> directed</span><span style=\"background-color: #FFF8F8\"> shot</span><span style=\"background-color: #FFF3F3\"> a</span><span style=\"background-color: #FFC4C4\"> really</span><span style=\"background-color: #FF0000\"> nice</span><span style=\"background-color: #FFFBFB\"> movie</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_6]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFF2F2\"> a</span><span style=\"background-color: #FFE5E5\"> wonderful</span><span style=\"background-color: #FF6767\"> film</span><span style=\"background-color: #FFA2A2\"> a</span><span style=\"background-color: #FFCECE\"> charming</span><span style=\"background-color: #FFEFEF\"> ,</span><span style=\"background-color: #FFE8E8\"> qui</span><span style=\"background-color: #FFCFCF\"> ##rky</span><span style=\"background-color: #FF0000\"> family</span><span style=\"background-color: #FFB6B6\"> story</span><span style=\"background-color: #FFFBFB\"> .</span><span style=\"background-color: #FFE9E9\"> a</span><span style=\"background-color: #FFA9A9\"> cross</span><span style=\"background-color: #FF5959\"> country</span><span style=\"background-color: #FFC2C2\"> journey</span><span style=\"background-color: #FFE9E9\"> filled</span><span style=\"background-color: #FFE7E7\"> with</span><span style=\"background-color: #FFD1D1\"> lots</span><span style=\"background-color: #FFD5D5\"> of</span><span style=\"background-color: #FFAAAA\"> interesting</span><span style=\"background-color: #FFEFEF\"> ,</span><span style=\"background-color: #FFD5D5\"> odd</span><span style=\"background-color: #FFD0D0\"> ##ball</span><span style=\"background-color: #FFE4E4\"> stops</span><span style=\"background-color: #FFFCFC\"> along</span><span style=\"background-color: #FFF4F4\"> the</span><span style=\"background-color: #FFFCFC\"> way</span><span style=\"background-color: #FFF0F0\"> several</span><span style=\"background-color: #FFC5C5\"> very</span><span style=\"background-color: #FFE7E7\"> cool</span><span style=\"background-color: #FFCECE\"> cameo</span><span style=\"background-color: #FFE9E9\"> ##s</span><span style=\"background-color: #FFF9F9\"> .</span><span style=\"background-color: #FFFCFC\"> great</span><span style=\"background-color: #FFCCCC\"> cast</span><span style=\"background-color: #FFDCDC\"> led</span><span style=\"background-color: #FFB7B7\"> by</span><span style=\"background-color: #FFF7F7\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFF8F8\"> ##iger</span><span style=\"background-color: #FFFBFB\"> carries</span><span style=\"background-color: #FFE0E0\"> the</span><span style=\"background-color: #FF6868\"> film</span><span style=\"background-color: #FFF4F4\"> along</span><span style=\"background-color: #FFC5C5\"> and</span><span style=\"background-color: #FFFCFC\"> leads</span><span style=\"background-color: #FFF9F9\"> to</span><span style=\"background-color: #FFE4E4\"> a</span><span style=\"background-color: #FFE5E5\"> surprise</span><span style=\"background-color: #FFAEAE\"> ending</span><span style=\"background-color: #FFF8F8\"> .</span><span style=\"background-color: #FFF1F1\"> well</span><span style=\"background-color: #FFF0F0\"> directed</span><span style=\"background-color: #FFDDDD\"> shot</span><span style=\"background-color: #FFE5E5\"> a</span><span style=\"background-color: #FFF5F5\"> really</span><span style=\"background-color: #FFF9F9\"> nice</span><span style=\"background-color: #FFD4D4\"> movie</span><span style=\"background-color: #FFF8F8\"> .</span><br><br>[BERTのAttentionを可視化_7]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FF0000\"> a</span><span style=\"background-color: #FF3939\"> wonderful</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFDBDB\"> a</span><span style=\"background-color: #FFA8A8\"> charming</span><span style=\"background-color: #FFF5F5\"> ,</span><span style=\"background-color: #FFFBFB\"> qui</span><span style=\"background-color: #FFF1F1\"> ##rky</span><span style=\"background-color: #FFFEFE\"> family</span><span style=\"background-color: #FFFCFC\"> story</span><span style=\"background-color: #FFE0E0\"> .</span><span style=\"background-color: #FFB9B9\"> a</span><span style=\"background-color: #FFFEFE\"> cross</span><span style=\"background-color: #FFFEFE\"> country</span><span style=\"background-color: #FFFEFE\"> journey</span><span style=\"background-color: #FFD6D6\"> filled</span><span style=\"background-color: #FFD9D9\"> with</span><span style=\"background-color: #FFD2D2\"> lots</span><span style=\"background-color: #FFEEEE\"> of</span><span style=\"background-color: #FFF2F2\"> interesting</span><span style=\"background-color: #FFF4F4\"> ,</span><span style=\"background-color: #FFFBFB\"> odd</span><span style=\"background-color: #FFF5F5\"> ##ball</span><span style=\"background-color: #FFFEFE\"> stops</span><span style=\"background-color: #FFF9F9\"> along</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> way</span><span style=\"background-color: #FFCECE\"> several</span><span style=\"background-color: #FFCDCD\"> very</span><span style=\"background-color: #FFEEEE\"> cool</span><span style=\"background-color: #FFFCFC\"> cameo</span><span style=\"background-color: #FFF8F8\"> ##s</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE6E6\"> great</span><span style=\"background-color: #FFE9E9\"> cast</span><span style=\"background-color: #FFFAFA\"> led</span><span style=\"background-color: #FFFBFB\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFEDED\"> carries</span><span style=\"background-color: #FFF9F9\"> the</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFAFA\"> along</span><span style=\"background-color: #FFE8E8\"> and</span><span style=\"background-color: #FFF0F0\"> leads</span><span style=\"background-color: #FFF6F6\"> to</span><span style=\"background-color: #FFD2D2\"> a</span><span style=\"background-color: #FFDFDF\"> surprise</span><span style=\"background-color: #FFFCFC\"> ending</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FF9B9B\"> well</span><span style=\"background-color: #FFFDFD\"> directed</span><span style=\"background-color: #FFFDFD\"> shot</span><span style=\"background-color: #FFE6E6\"> a</span><span style=\"background-color: #FFEDED\"> really</span><span style=\"background-color: #FFEDED\"> nice</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_8]<br><span style=\"background-color: #FFFDFD\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> wonderful</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> charming</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> qui</span><span style=\"background-color: #FFFEFE\"> ##rky</span><span style=\"background-color: #FFF3F3\"> family</span><span style=\"background-color: #FFFEFE\"> story</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFF8F8\"> cross</span><span style=\"background-color: #FF5C5C\"> country</span><span style=\"background-color: #FFFEFE\"> journey</span><span style=\"background-color: #FFFEFE\"> filled</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFEFE\"> lots</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> odd</span><span style=\"background-color: #FFFEFE\"> ##ball</span><span style=\"background-color: #FFFEFE\"> stops</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> way</span><span style=\"background-color: #FFFEFE\"> several</span><span style=\"background-color: #FFFEFE\"> very</span><span style=\"background-color: #FFFEFE\"> cool</span><span style=\"background-color: #FFFEFE\"> cameo</span><span style=\"background-color: #FFFEFE\"> ##s</span><span style=\"background-color: #FF5F5F\"> .</span><span style=\"background-color: #FFFEFE\"> great</span><span style=\"background-color: #FFFDFD\"> cast</span><span style=\"background-color: #FFFEFE\"> led</span><span style=\"background-color: #FFFDFD\"> by</span><span style=\"background-color: #FFD8D8\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFF9F9\"> ##iger</span><span style=\"background-color: #FFFEFE\"> carries</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> leads</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> surprise</span><span style=\"background-color: #FFFEFE\"> ending</span><span style=\"background-color: #FF1A1A\"> .</span><span style=\"background-color: #FFFEFE\"> well</span><span style=\"background-color: #FFFEFE\"> directed</span><span style=\"background-color: #FFFCFC\"> shot</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> really</span><span style=\"background-color: #FFFDFD\"> nice</span><span style=\"background-color: #FFFBFB\"> movie</span><span style=\"background-color: #FF2828\"> .</span><br><br>[BERTのAttentionを可視化_9]<br><span style=\"background-color: #FFF4F4\"> [CLS]</span><span style=\"background-color: #FFEDED\"> a</span><span style=\"background-color: #FFABAB\"> wonderful</span><span style=\"background-color: #FFF9F9\"> film</span><span style=\"background-color: #FFE7E7\"> a</span><span style=\"background-color: #FFE6E6\"> charming</span><span style=\"background-color: #FFEDED\"> ,</span><span style=\"background-color: #FFFDFD\"> qui</span><span style=\"background-color: #FFF6F6\"> ##rky</span><span style=\"background-color: #FFF8F8\"> family</span><span style=\"background-color: #FFF8F8\"> story</span><span style=\"background-color: #FF9191\"> .</span><span style=\"background-color: #FFEDED\"> a</span><span style=\"background-color: #FFFDFD\"> cross</span><span style=\"background-color: #FFF9F9\"> country</span><span style=\"background-color: #FFF2F2\"> journey</span><span style=\"background-color: #FFE5E5\"> filled</span><span style=\"background-color: #FFFAFA\"> with</span><span style=\"background-color: #FFF4F4\"> lots</span><span style=\"background-color: #FFFBFB\"> of</span><span style=\"background-color: #FFFCFC\"> interesting</span><span style=\"background-color: #FFF1F1\"> ,</span><span style=\"background-color: #FFFEFE\"> odd</span><span style=\"background-color: #FFF8F8\"> ##ball</span><span style=\"background-color: #FFF9F9\"> stops</span><span style=\"background-color: #FFFCFC\"> along</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFDDDD\"> way</span><span style=\"background-color: #FFFAFA\"> several</span><span style=\"background-color: #FFEDED\"> very</span><span style=\"background-color: #FFF9F9\"> cool</span><span style=\"background-color: #FFFDFD\"> cameo</span><span style=\"background-color: #FFFCFC\"> ##s</span><span style=\"background-color: #FFFAFA\"> .</span><span style=\"background-color: #FFF5F5\"> great</span><span style=\"background-color: #FFCBCB\"> cast</span><span style=\"background-color: #FFCECE\"> led</span><span style=\"background-color: #FFF5F5\"> by</span><span style=\"background-color: #FFFCFC\"> rod</span><span style=\"background-color: #FFFCFC\"> ste</span><span style=\"background-color: #FFFBFB\"> ##iger</span><span style=\"background-color: #FF1D1D\"> carries</span><span style=\"background-color: #FFF9F9\"> the</span><span style=\"background-color: #FFF9F9\"> film</span><span style=\"background-color: #FFD0D0\"> along</span><span style=\"background-color: #FF5151\"> and</span><span style=\"background-color: #FFD3D3\"> leads</span><span style=\"background-color: #FFF6F6\"> to</span><span style=\"background-color: #FFE4E4\"> a</span><span style=\"background-color: #FFD2D2\"> surprise</span><span style=\"background-color: #FFF7F7\"> ending</span><span style=\"background-color: #FFFAFA\"> .</span><span style=\"background-color: #FF9191\"> well</span><span style=\"background-color: #FFE5E5\"> directed</span><span style=\"background-color: #FFC2C2\"> shot</span><span style=\"background-color: #FFDFDF\"> a</span><span style=\"background-color: #FFC8C8\"> really</span><span style=\"background-color: #FF0000\"> nice</span><span style=\"background-color: #FFF3F3\"> movie</span><span style=\"background-color: #FFFAFA\"> .</span><br><br>[BERTのAttentionを可視化_10]<br><span style=\"background-color: #FFEBEB\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFDFD\"> wonderful</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> charming</span><span style=\"background-color: #FFF5F5\"> ,</span><span style=\"background-color: #FFFEFE\"> qui</span><span style=\"background-color: #FFFEFE\"> ##rky</span><span style=\"background-color: #FFFDFD\"> family</span><span style=\"background-color: #FFFEFE\"> story</span><span style=\"background-color: #FFF6F6\"> .</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> cross</span><span style=\"background-color: #FFFEFE\"> country</span><span style=\"background-color: #FFFCFC\"> journey</span><span style=\"background-color: #FFF8F8\"> filled</span><span style=\"background-color: #FFE9E9\"> with</span><span style=\"background-color: #FFF4F4\"> lots</span><span style=\"background-color: #FFF9F9\"> of</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFF0F0\"> ,</span><span style=\"background-color: #FFFEFE\"> odd</span><span style=\"background-color: #FFFBFB\"> ##ball</span><span style=\"background-color: #FFFEFE\"> stops</span><span style=\"background-color: #FFF9F9\"> along</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFAFA\"> way</span><span style=\"background-color: #FFFDFD\"> several</span><span style=\"background-color: #FFF5F5\"> very</span><span style=\"background-color: #FFFDFD\"> cool</span><span style=\"background-color: #FFFEFE\"> cameo</span><span style=\"background-color: #FFFEFE\"> ##s</span><span style=\"background-color: #FFF8F8\"> .</span><span style=\"background-color: #FFFEFE\"> great</span><span style=\"background-color: #FFFCFC\"> cast</span><span style=\"background-color: #FFEFEF\"> led</span><span style=\"background-color: #FFFCFC\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFFDFD\"> carries</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFE4E4\"> along</span><span style=\"background-color: #FFB8B8\"> and</span><span style=\"background-color: #FFFDFD\"> leads</span><span style=\"background-color: #FFE6E6\"> to</span><span style=\"background-color: #FFF8F8\"> a</span><span style=\"background-color: #FFFBFB\"> surprise</span><span style=\"background-color: #FFFBFB\"> ending</span><span style=\"background-color: #FFF7F7\"> .</span><span style=\"background-color: #FFD4D4\"> well</span><span style=\"background-color: #FFFBFB\"> directed</span><span style=\"background-color: #FFF5F5\"> shot</span><span style=\"background-color: #FFF3F3\"> a</span><span style=\"background-color: #FF0000\"> really</span><span style=\"background-color: #FFF5F5\"> nice</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFF7F7\"> .</span><br><br>[BERTのAttentionを可視化_11]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFEBEB\"> a</span><span style=\"background-color: #FF9D9D\"> wonderful</span><span style=\"background-color: #FFFDFD\"> film</span><span style=\"background-color: #FFFCFC\"> a</span><span style=\"background-color: #FFD5D5\"> charming</span><span style=\"background-color: #FFF0F0\"> ,</span><span style=\"background-color: #FFFAFA\"> qui</span><span style=\"background-color: #FFFAFA\"> ##rky</span><span style=\"background-color: #FFFEFE\"> family</span><span style=\"background-color: #FFFBFB\"> story</span><span style=\"background-color: #FFFAFA\"> .</span><span style=\"background-color: #FFF0F0\"> a</span><span style=\"background-color: #FFFEFE\"> cross</span><span style=\"background-color: #FFFEFE\"> country</span><span style=\"background-color: #FFFCFC\"> journey</span><span style=\"background-color: #FFFCFC\"> filled</span><span style=\"background-color: #FFFBFB\"> with</span><span style=\"background-color: #FFA8A8\"> lots</span><span style=\"background-color: #FFF6F6\"> of</span><span style=\"background-color: #FFF9F9\"> interesting</span><span style=\"background-color: #FFEEEE\"> ,</span><span style=\"background-color: #FFF3F3\"> odd</span><span style=\"background-color: #FFF4F4\"> ##ball</span><span style=\"background-color: #FFFBFB\"> stops</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFFDFD\"> way</span><span style=\"background-color: #FFFDFD\"> several</span><span style=\"background-color: #FF0000\"> very</span><span style=\"background-color: #FFF9F9\"> cool</span><span style=\"background-color: #FFFBFB\"> cameo</span><span style=\"background-color: #FFFDFD\"> ##s</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFF2F2\"> great</span><span style=\"background-color: #FFFBFB\"> cast</span><span style=\"background-color: #FFFEFE\"> led</span><span style=\"background-color: #FFFEFE\"> by</span><span style=\"background-color: #FFFEFE\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFFEFE\"> ##iger</span><span style=\"background-color: #FFD8D8\"> carries</span><span style=\"background-color: #FFF7F7\"> the</span><span style=\"background-color: #FFFCFC\"> film</span><span style=\"background-color: #FFFEFE\"> along</span><span style=\"background-color: #FFE7E7\"> and</span><span style=\"background-color: #FFF4F4\"> leads</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFF5F5\"> a</span><span style=\"background-color: #FFDBDB\"> surprise</span><span style=\"background-color: #FFFEFE\"> ending</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FF6262\"> well</span><span style=\"background-color: #FFF5F5\"> directed</span><span style=\"background-color: #FFFCFC\"> shot</span><span style=\"background-color: #FFF8F8\"> a</span><span style=\"background-color: #FFC9C9\"> really</span><span style=\"background-color: #FFE3E3\"> nice</span><span style=\"background-color: #FFFDFD\"> movie</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_12]<br><span style=\"background-color: #FFF2F2\"> [CLS]</span><span style=\"background-color: #FFC5C5\"> a</span><span style=\"background-color: #FFADAD\"> wonderful</span><span style=\"background-color: #FFDDDD\"> film</span><span style=\"background-color: #FFD5D5\"> a</span><span style=\"background-color: #FFBDBD\"> charming</span><span style=\"background-color: #FFE9E9\"> ,</span><span style=\"background-color: #FFF6F6\"> qui</span><span style=\"background-color: #FFEDED\"> ##rky</span><span style=\"background-color: #FFDFDF\"> family</span><span style=\"background-color: #FFF0F0\"> story</span><span style=\"background-color: #FFECEC\"> .</span><span style=\"background-color: #FFEFEF\"> a</span><span style=\"background-color: #FFF6F6\"> cross</span><span style=\"background-color: #FFE7E7\"> country</span><span style=\"background-color: #FFF3F3\"> journey</span><span style=\"background-color: #FFFAFA\"> filled</span><span style=\"background-color: #FFFCFC\"> with</span><span style=\"background-color: #FFFCFC\"> lots</span><span style=\"background-color: #FFFAFA\"> of</span><span style=\"background-color: #FFF0F0\"> interesting</span><span style=\"background-color: #FFF8F8\"> ,</span><span style=\"background-color: #FFFAFA\"> odd</span><span style=\"background-color: #FFFAFA\"> ##ball</span><span style=\"background-color: #FFFAFA\"> stops</span><span style=\"background-color: #FFFDFD\"> along</span><span style=\"background-color: #FFF7F7\"> the</span><span style=\"background-color: #FFFBFB\"> way</span><span style=\"background-color: #FFFEFE\"> several</span><span style=\"background-color: #FFF1F1\"> very</span><span style=\"background-color: #FFF7F7\"> cool</span><span style=\"background-color: #FFFCFC\"> cameo</span><span style=\"background-color: #FFFEFE\"> ##s</span><span style=\"background-color: #FF9292\"> .</span><span style=\"background-color: #FFEEEE\"> great</span><span style=\"background-color: #FFEBEB\"> cast</span><span style=\"background-color: #FFFAFA\"> led</span><span style=\"background-color: #FFF0F0\"> by</span><span style=\"background-color: #FFFAFA\"> rod</span><span style=\"background-color: #FFFEFE\"> ste</span><span style=\"background-color: #FFF5F5\"> ##iger</span><span style=\"background-color: #FFF5F5\"> carries</span><span style=\"background-color: #FFADAD\"> the</span><span style=\"background-color: #FFB4B4\"> film</span><span style=\"background-color: #FFF3F3\"> along</span><span style=\"background-color: #FFE8E8\"> and</span><span style=\"background-color: #FFFAFA\"> leads</span><span style=\"background-color: #FFF0F0\"> to</span><span style=\"background-color: #FFE0E0\"> a</span><span style=\"background-color: #FFA3A3\"> surprise</span><span style=\"background-color: #FFEEEE\"> ending</span><span style=\"background-color: #FF8484\"> .</span><span style=\"background-color: #FFB9B9\"> well</span><span style=\"background-color: #FFE4E4\"> directed</span><span style=\"background-color: #FFD9D9\"> shot</span><span style=\"background-color: #FF4545\"> a</span><span style=\"background-color: #FF7171\"> really</span><span style=\"background-color: #FF0000\"> nice</span><span style=\"background-color: #FF7878\"> movie</span><span style=\"background-color: #FF8383\"> .</span><br><br>[BERTのAttentionを可視化_ALL]<br><span style=\"background-color: #FFECEC\"> [CLS]</span><span style=\"background-color: #FF0000\"> a</span><span style=\"background-color: #FF5555\"> wonderful</span><span style=\"background-color: #FF9494\"> film</span><span style=\"background-color: #FF7676\"> a</span><span style=\"background-color: #FFA6A6\"> charming</span><span style=\"background-color: #FFC5C5\"> ,</span><span style=\"background-color: #FFEFEF\"> qui</span><span style=\"background-color: #FFDFDF\"> ##rky</span><span style=\"background-color: #FFA9A9\"> family</span><span style=\"background-color: #FFD5D5\"> story</span><span style=\"background-color: #FFB6B6\"> .</span><span style=\"background-color: #FF8686\"> a</span><span style=\"background-color: #FFD5D5\"> cross</span><span style=\"background-color: #FF9B9B\"> country</span><span style=\"background-color: #FFD2D2\"> journey</span><span style=\"background-color: #FFD5D5\"> filled</span><span style=\"background-color: #FFD0D0\"> with</span><span style=\"background-color: #FFC2C2\"> lots</span><span style=\"background-color: #FFD9D9\"> of</span><span style=\"background-color: #FFCBCB\"> interesting</span><span style=\"background-color: #FFC0C0\"> ,</span><span style=\"background-color: #FFD2D2\"> odd</span><span style=\"background-color: #FFD3D3\"> ##ball</span><span style=\"background-color: #FFADAD\"> stops</span><span style=\"background-color: #FFD6D6\"> along</span><span style=\"background-color: #FFEDED\"> the</span><span style=\"background-color: #FFE5E5\"> way</span><span style=\"background-color: #FFD5D5\"> several</span><span style=\"background-color: #FF8787\"> very</span><span style=\"background-color: #FFB7B7\"> cool</span><span style=\"background-color: #FFE1E1\"> cameo</span><span style=\"background-color: #FFEAEA\"> ##s</span><span style=\"background-color: #FF8686\"> .</span><span style=\"background-color: #FFDFDF\"> great</span><span style=\"background-color: #FF9191\"> cast</span><span style=\"background-color: #FFCACA\"> led</span><span style=\"background-color: #FFD6D6\"> by</span><span style=\"background-color: #FFEDED\"> rod</span><span style=\"background-color: #FFFDFD\"> ste</span><span style=\"background-color: #FFF6F6\"> ##iger</span><span style=\"background-color: #FF9595\"> carries</span><span style=\"background-color: #FFB3B3\"> the</span><span style=\"background-color: #FF6E6E\"> film</span><span style=\"background-color: #FFC0C0\"> along</span><span style=\"background-color: #FF8282\"> and</span><span style=\"background-color: #FFC8C8\"> leads</span><span style=\"background-color: #FFD7D7\"> to</span><span style=\"background-color: #FFA3A3\"> a</span><span style=\"background-color: #FFB5B5\"> surprise</span><span style=\"background-color: #FFACAC\"> ending</span><span style=\"background-color: #FF6767\"> .</span><span style=\"background-color: #FF3939\"> well</span><span style=\"background-color: #FFC8C8\"> directed</span><span style=\"background-color: #FFC1C1\"> shot</span><span style=\"background-color: #FF6B6B\"> a</span><span style=\"background-color: #FF6060\"> really</span><span style=\"background-color: #FF1A1A\"> nice</span><span style=\"background-color: #FF8989\"> movie</span><span style=\"background-color: #FF6E6E\"> .</span><br><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "index = 3  # 出力させたいデータ\n",
    "html_output = mk_html(index, batch, preds, attention_probs, TEXT)  # HTML作成\n",
    "HTML(html_output)  # HTML形式で出力\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "正解ラベル：Positive<br>推論ラベル：Positive<br><br>[BERTのAttentionを可視化_1]<br><span style=\"background-color: #FFF9F9\"> [CLS]</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFDADA\"> watched</span><span style=\"background-color: #FFE9E9\"> this</span><span style=\"background-color: #FFECEC\"> movie</span><span style=\"background-color: #FFF8F8\"> as</span><span style=\"background-color: #FFFAFA\"> a</span><span style=\"background-color: #FFFCFC\"> child</span><span style=\"background-color: #FFF3F3\"> and</span><span style=\"background-color: #FFEFEF\"> still</span><span style=\"background-color: #FFEDED\"> enjoy</span><span style=\"background-color: #FF6B6B\"> viewing</span><span style=\"background-color: #FFD8D8\"> it</span><span style=\"background-color: #FFECEC\"> every</span><span style=\"background-color: #FFFBFB\"> once</span><span style=\"background-color: #FFF5F5\"> in</span><span style=\"background-color: #FFF9F9\"> a</span><span style=\"background-color: #FFE9E9\"> while</span><span style=\"background-color: #FFA9A9\"> for</span><span style=\"background-color: #FF8080\"> the</span><span style=\"background-color: #FF9292\"> nostalgia</span><span style=\"background-color: #FFA4A4\"> factor</span><span style=\"background-color: #FF9393\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFDFD\"> younger</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFF9F9\"> loved</span><span style=\"background-color: #FFFAFA\"> the</span><span style=\"background-color: #FFEDED\"> movie</span><span style=\"background-color: #FFF0F0\"> because</span><span style=\"background-color: #FFF8F8\"> of</span><span style=\"background-color: #FFE9E9\"> the</span><span style=\"background-color: #FFE7E7\"> entertaining</span><span style=\"background-color: #FFD1D1\"> storyline</span><span style=\"background-color: #FFE9E9\"> and</span><span style=\"background-color: #FFEBEB\"> interesting</span><span style=\"background-color: #FFEEEE\"> characters</span><span style=\"background-color: #FF9F9F\"> .</span><span style=\"background-color: #FFF1F1\"> today</span><span style=\"background-color: #FFF4F4\"> ,</span><span style=\"background-color: #FFEFEF\"> i</span><span style=\"background-color: #FFF7F7\"> still</span><span style=\"background-color: #FFCCCC\"> love</span><span style=\"background-color: #FFF6F6\"> the</span><span style=\"background-color: #FFEFEF\"> characters</span><span style=\"background-color: #FF9393\"> .</span><span style=\"background-color: #FFFCFC\"> additionally</span><span style=\"background-color: #FFF7F7\"> ,</span><span style=\"background-color: #FFDFDF\"> i</span><span style=\"background-color: #FF0000\"> think</span><span style=\"background-color: #FFF5F5\"> of</span><span style=\"background-color: #FFF4F4\"> the</span><span style=\"background-color: #FFE8E8\"> plot</span><span style=\"background-color: #FFF0F0\"> with</span><span style=\"background-color: #FFF6F6\"> higher</span><span style=\"background-color: #FFB0B0\"> regard</span><span style=\"background-color: #FFF2F2\"> because</span><span style=\"background-color: #FFE7E7\"> i</span><span style=\"background-color: #FFF7F7\"> now</span><span style=\"background-color: #FFBBBB\"> see</span><span style=\"background-color: #FFDBDB\"> the</span><span style=\"background-color: #FF7F7F\"> morals</span><span style=\"background-color: #FFC5C5\"> and</span><span style=\"background-color: #FFB2B2\"> symbolism</span><span style=\"background-color: #FF9393\"> .</span><span style=\"background-color: #FFFDFD\"> rainbow</span><span style=\"background-color: #FFFAFA\"> brit</span><span style=\"background-color: #FFFDFD\"> ##e</span><span style=\"background-color: #FFF1F1\"> is</span><span style=\"background-color: #FFFAFA\"> far</span><span style=\"background-color: #FFE1E1\"> from</span><span style=\"background-color: #FFF2F2\"> the</span><span style=\"background-color: #FFCECE\"> worst</span><span style=\"background-color: #FFF8F8\"> film</span><span style=\"background-color: #FFFDFD\"> ever</span><span style=\"background-color: #FFEAEA\"> ,</span><span style=\"background-color: #FFF5F5\"> and</span><span style=\"background-color: #FFEEEE\"> though</span><span style=\"background-color: #FFEAEA\"> out</span><span style=\"background-color: #FFC2C2\"> dated</span><span style=\"background-color: #FFF2F2\"> ,</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFF7F7\"> m</span><span style=\"background-color: #FFF6F6\"> sure</span><span style=\"background-color: #FFF3F3\"> i</span><span style=\"background-color: #FFFCFC\"> will</span><span style=\"background-color: #FF8080\"> show</span><span style=\"background-color: #FFDADA\"> it</span><span style=\"background-color: #FFF6F6\"> to</span><span style=\"background-color: #FFFCFC\"> my</span><span style=\"background-color: #FFFBFB\"> children</span><span style=\"background-color: #FFFCFC\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFCFC\"> future</span><span style=\"background-color: #FFE0E0\"> ,</span><span style=\"background-color: #FFF8F8\"> when</span><span style=\"background-color: #FFFAFA\"> i</span><span style=\"background-color: #FFF7F7\"> have</span><span style=\"background-color: #FFF8F8\"> children</span><span style=\"background-color: #FF9292\"> .</span><br><br>[BERTのAttentionを可視化_2]<br><span style=\"background-color: #FFC5C5\"> [CLS]</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFF9F9\"> watched</span><span style=\"background-color: #FFF4F4\"> this</span><span style=\"background-color: #FF9393\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFDFD\"> a</span><span style=\"background-color: #FFFDFD\"> child</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFDFD\"> enjoy</span><span style=\"background-color: #FFF4F4\"> viewing</span><span style=\"background-color: #FFFAFA\"> it</span><span style=\"background-color: #FFFEFE\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFFEFE\"> for</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFF7F7\"> nostalgia</span><span style=\"background-color: #FFFEFE\"> factor</span><span style=\"background-color: #FFD6D6\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFCFC\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFF5F5\"> the</span><span style=\"background-color: #FFDADA\"> movie</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFDFD\"> of</span><span style=\"background-color: #FFF8F8\"> the</span><span style=\"background-color: #FFEFEF\"> entertaining</span><span style=\"background-color: #FFE6E6\"> storyline</span><span style=\"background-color: #FFFBFB\"> and</span><span style=\"background-color: #FFF3F3\"> interesting</span><span style=\"background-color: #FFEEEE\"> characters</span><span style=\"background-color: #FFDADA\"> .</span><span style=\"background-color: #FFFEFE\"> today</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFF2F2\"> i</span><span style=\"background-color: #FFFDFD\"> still</span><span style=\"background-color: #FFE9E9\"> love</span><span style=\"background-color: #FFECEC\"> the</span><span style=\"background-color: #FFF4F4\"> characters</span><span style=\"background-color: #FFD5D5\"> .</span><span style=\"background-color: #FFFDFD\"> additionally</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFF4F4\"> i</span><span style=\"background-color: #FFFAFA\"> think</span><span style=\"background-color: #FFE9E9\"> of</span><span style=\"background-color: #FFB9B9\"> the</span><span style=\"background-color: #FFDEDE\"> plot</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFEFE\"> higher</span><span style=\"background-color: #FFFEFE\"> regard</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFDFD\"> now</span><span style=\"background-color: #FFFBFB\"> see</span><span style=\"background-color: #FFE9E9\"> the</span><span style=\"background-color: #FFF8F8\"> morals</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFCFC\"> symbolism</span><span style=\"background-color: #FFD5D5\"> .</span><span style=\"background-color: #FFF1F1\"> rainbow</span><span style=\"background-color: #FFF8F8\"> brit</span><span style=\"background-color: #FFBFBF\"> ##e</span><span style=\"background-color: #FFC9C9\"> is</span><span style=\"background-color: #FFFDFD\"> far</span><span style=\"background-color: #FFFCFC\"> from</span><span style=\"background-color: #FFDADA\"> the</span><span style=\"background-color: #FFF4F4\"> worst</span><span style=\"background-color: #FF0000\"> film</span><span style=\"background-color: #FFF7F7\"> ever</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFFBFB\"> and</span><span style=\"background-color: #FFFEFE\"> though</span><span style=\"background-color: #FFF9F9\"> out</span><span style=\"background-color: #FFF1F1\"> dated</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFECEC\"> i</span><span style=\"background-color: #FFFBFB\"> m</span><span style=\"background-color: #FFF8F8\"> sure</span><span style=\"background-color: #FFEDED\"> i</span><span style=\"background-color: #FFF3F3\"> will</span><span style=\"background-color: #FFA5A5\"> show</span><span style=\"background-color: #FFE2E2\"> it</span><span style=\"background-color: #FFEEEE\"> to</span><span style=\"background-color: #FFEDED\"> my</span><span style=\"background-color: #FFF3F3\"> children</span><span style=\"background-color: #FFFBFB\"> in</span><span style=\"background-color: #FFEFEF\"> the</span><span style=\"background-color: #FFF8F8\"> future</span><span style=\"background-color: #FFF3F3\"> ,</span><span style=\"background-color: #FFFCFC\"> when</span><span style=\"background-color: #FFD6D6\"> i</span><span style=\"background-color: #FFF0F0\"> have</span><span style=\"background-color: #FFE1E1\"> children</span><span style=\"background-color: #FFD5D5\"> .</span><br><br>[BERTのAttentionを可視化_3]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFCFC\"> i</span><span style=\"background-color: #FFFEFE\"> watched</span><span style=\"background-color: #FFCBCB\"> this</span><span style=\"background-color: #FFFAFA\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> child</span><span style=\"background-color: #FFF7F7\"> and</span><span style=\"background-color: #FFFBFB\"> still</span><span style=\"background-color: #FFEFEF\"> enjoy</span><span style=\"background-color: #FFFBFB\"> viewing</span><span style=\"background-color: #FFF7F7\"> it</span><span style=\"background-color: #FFF8F8\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFCFC\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFF3F3\"> for</span><span style=\"background-color: #FFD6D6\"> the</span><span style=\"background-color: #FFFCFC\"> nostalgia</span><span style=\"background-color: #FFFCFC\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFFDFD\"> loved</span><span style=\"background-color: #FFF1F1\"> the</span><span style=\"background-color: #FFFDFD\"> movie</span><span style=\"background-color: #FFFDFD\"> because</span><span style=\"background-color: #FFF9F9\"> of</span><span style=\"background-color: #FFB7B7\"> the</span><span style=\"background-color: #FFE0E0\"> entertaining</span><span style=\"background-color: #FFEFEF\"> storyline</span><span style=\"background-color: #FFE0E0\"> and</span><span style=\"background-color: #FFE3E3\"> interesting</span><span style=\"background-color: #FFFCFC\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFF8F8\"> today</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFE6E6\"> i</span><span style=\"background-color: #FFF8F8\"> still</span><span style=\"background-color: #FFBDBD\"> love</span><span style=\"background-color: #FFE8E8\"> the</span><span style=\"background-color: #FFFCFC\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE5E5\"> additionally</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFEDED\"> i</span><span style=\"background-color: #FFFDFD\"> think</span><span style=\"background-color: #FFF5F5\"> of</span><span style=\"background-color: #FF2828\"> the</span><span style=\"background-color: #FFEAEA\"> plot</span><span style=\"background-color: #FFEBEB\"> with</span><span style=\"background-color: #FFFDFD\"> higher</span><span style=\"background-color: #FFFDFD\"> regard</span><span style=\"background-color: #FFF3F3\"> because</span><span style=\"background-color: #FFF6F6\"> i</span><span style=\"background-color: #FFF6F6\"> now</span><span style=\"background-color: #FFE2E2\"> see</span><span style=\"background-color: #FFA1A1\"> the</span><span style=\"background-color: #FFFBFB\"> morals</span><span style=\"background-color: #FFF7F7\"> and</span><span style=\"background-color: #FFF1F1\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFF3F3\"> rainbow</span><span style=\"background-color: #FFFBFB\"> brit</span><span style=\"background-color: #FFDFDF\"> ##e</span><span style=\"background-color: #FF0000\"> is</span><span style=\"background-color: #FFF6F6\"> far</span><span style=\"background-color: #FF9292\"> from</span><span style=\"background-color: #FF9A9A\"> the</span><span style=\"background-color: #FFF4F4\"> worst</span><span style=\"background-color: #FFF4F4\"> film</span><span style=\"background-color: #FFFBFB\"> ever</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FF1C1C\"> and</span><span style=\"background-color: #FFF4F4\"> though</span><span style=\"background-color: #FFFDFD\"> out</span><span style=\"background-color: #FFFBFB\"> dated</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFE4E4\"> i</span><span style=\"background-color: #FFF7F7\"> m</span><span style=\"background-color: #FFF0F0\"> sure</span><span style=\"background-color: #FFCFCF\"> i</span><span style=\"background-color: #FFFAFA\"> will</span><span style=\"background-color: #FFCACA\"> show</span><span style=\"background-color: #FFCCCC\"> it</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFE4E4\"> my</span><span style=\"background-color: #FFFDFD\"> children</span><span style=\"background-color: #FFFCFC\"> in</span><span style=\"background-color: #FFF0F0\"> the</span><span style=\"background-color: #FFFDFD\"> future</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFFDFD\"> when</span><span style=\"background-color: #FFA7A7\"> i</span><span style=\"background-color: #FFFDFD\"> have</span><span style=\"background-color: #FFFBFB\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_4]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFF5F5\"> watched</span><span style=\"background-color: #FFC5C5\"> this</span><span style=\"background-color: #FFE3E3\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> child</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFEFE\"> enjoy</span><span style=\"background-color: #FFFAFA\"> viewing</span><span style=\"background-color: #FFB1B1\"> it</span><span style=\"background-color: #FFFEFE\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFFEFE\"> for</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> nostalgia</span><span style=\"background-color: #FFFEFE\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFDFD\"> loved</span><span style=\"background-color: #FFB3B3\"> the</span><span style=\"background-color: #FFE4E4\"> movie</span><span style=\"background-color: #FFFBFB\"> because</span><span style=\"background-color: #FFECEC\"> of</span><span style=\"background-color: #FF4E4E\"> the</span><span style=\"background-color: #FFEEEE\"> entertaining</span><span style=\"background-color: #FFEFEF\"> storyline</span><span style=\"background-color: #FFFCFC\"> and</span><span style=\"background-color: #FFF8F8\"> interesting</span><span style=\"background-color: #FFFCFC\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> today</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFDFD\"> love</span><span style=\"background-color: #FFD9D9\"> the</span><span style=\"background-color: #FFFCFC\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> additionally</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> think</span><span style=\"background-color: #FFFCFC\"> of</span><span style=\"background-color: #FF3232\"> the</span><span style=\"background-color: #FFE7E7\"> plot</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFEFE\"> higher</span><span style=\"background-color: #FFFEFE\"> regard</span><span style=\"background-color: #FFFDFD\"> because</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> now</span><span style=\"background-color: #FFFBFB\"> see</span><span style=\"background-color: #FF4D4D\"> the</span><span style=\"background-color: #FFF9F9\"> morals</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFF8F8\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFCFC\"> rainbow</span><span style=\"background-color: #FFFDFD\"> brit</span><span style=\"background-color: #FFF7F7\"> ##e</span><span style=\"background-color: #FF0B0B\"> is</span><span style=\"background-color: #FFFCFC\"> far</span><span style=\"background-color: #FFFCFC\"> from</span><span style=\"background-color: #FF0000\"> the</span><span style=\"background-color: #FFEFEF\"> worst</span><span style=\"background-color: #FFDADA\"> film</span><span style=\"background-color: #FFF9F9\"> ever</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFF9F9\"> and</span><span style=\"background-color: #FFFCFC\"> though</span><span style=\"background-color: #FFFDFD\"> out</span><span style=\"background-color: #FFFDFD\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFFDFD\"> m</span><span style=\"background-color: #FFFEFE\"> sure</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFFEFE\"> will</span><span style=\"background-color: #FFF9F9\"> show</span><span style=\"background-color: #FF2626\"> it</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFDFD\"> my</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> future</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_5]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> watched</span><span style=\"background-color: #FFFEFE\"> this</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> child</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFDFD\"> still</span><span style=\"background-color: #FFFDFD\"> enjoy</span><span style=\"background-color: #FFFEFE\"> viewing</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFEFE\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFFEFE\"> for</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> nostalgia</span><span style=\"background-color: #FFFEFE\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> entertaining</span><span style=\"background-color: #FFFEFE\"> storyline</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFCFC\"> today</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFF3F3\"> love</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> additionally</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> think</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> plot</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFEFE\"> higher</span><span style=\"background-color: #FFFEFE\"> regard</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> now</span><span style=\"background-color: #FFFEFE\"> see</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> morals</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFEFE\"> ##e</span><span style=\"background-color: #FFFEFE\"> is</span><span style=\"background-color: #FFFBFB\"> far</span><span style=\"background-color: #FFFEFE\"> from</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FF0000\"> worst</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> ever</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> though</span><span style=\"background-color: #FFFEFE\"> out</span><span style=\"background-color: #FFFEFE\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> m</span><span style=\"background-color: #FFFEFE\"> sure</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> will</span><span style=\"background-color: #FFFEFE\"> show</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFEFE\"> my</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> future</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_6]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFF7F7\"> watched</span><span style=\"background-color: #FFD6D6\"> this</span><span style=\"background-color: #FFD7D7\"> movie</span><span style=\"background-color: #FFFAFA\"> as</span><span style=\"background-color: #FFF6F6\"> a</span><span style=\"background-color: #FFF5F5\"> child</span><span style=\"background-color: #FFBFBF\"> and</span><span style=\"background-color: #FFD6D6\"> still</span><span style=\"background-color: #FFF3F3\"> enjoy</span><span style=\"background-color: #FFE3E3\"> viewing</span><span style=\"background-color: #FFF7F7\"> it</span><span style=\"background-color: #FF7676\"> every</span><span style=\"background-color: #FFEFEF\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFAFA\"> a</span><span style=\"background-color: #FFF7F7\"> while</span><span style=\"background-color: #FFFBFB\"> for</span><span style=\"background-color: #FF9E9E\"> the</span><span style=\"background-color: #FFB4B4\"> nostalgia</span><span style=\"background-color: #FF7272\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFDFD\"> when</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFDFD\"> was</span><span style=\"background-color: #FFFAFA\"> younger</span><span style=\"background-color: #FFF9F9\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFFBFB\"> the</span><span style=\"background-color: #FFE6E6\"> movie</span><span style=\"background-color: #FFF7F7\"> because</span><span style=\"background-color: #FFFDFD\"> of</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFCFC\"> entertaining</span><span style=\"background-color: #FFE2E2\"> storyline</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFF9F9\"> interesting</span><span style=\"background-color: #FFE7E7\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FF4F4F\"> today</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFF3F3\"> i</span><span style=\"background-color: #FFE6E6\"> still</span><span style=\"background-color: #FFF2F2\"> love</span><span style=\"background-color: #FFF9F9\"> the</span><span style=\"background-color: #FFE1E1\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFEEEE\"> additionally</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFFAFA\"> think</span><span style=\"background-color: #FFEDED\"> of</span><span style=\"background-color: #FFF2F2\"> the</span><span style=\"background-color: #FFEEEE\"> plot</span><span style=\"background-color: #FFFDFD\"> with</span><span style=\"background-color: #FFFBFB\"> higher</span><span style=\"background-color: #FFFDFD\"> regard</span><span style=\"background-color: #FFD4D4\"> because</span><span style=\"background-color: #FFF8F8\"> i</span><span style=\"background-color: #FFDBDB\"> now</span><span style=\"background-color: #FFF9F9\"> see</span><span style=\"background-color: #FFF6F6\"> the</span><span style=\"background-color: #FFEAEA\"> morals</span><span style=\"background-color: #FFF3F3\"> and</span><span style=\"background-color: #FFF6F6\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFCFC\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFDFD\"> ##e</span><span style=\"background-color: #FFF3F3\"> is</span><span style=\"background-color: #FFF9F9\"> far</span><span style=\"background-color: #FFFCFC\"> from</span><span style=\"background-color: #FFF4F4\"> the</span><span style=\"background-color: #FFD0D0\"> worst</span><span style=\"background-color: #FFD2D2\"> film</span><span style=\"background-color: #FFF1F1\"> ever</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FFF3F3\"> and</span><span style=\"background-color: #FFDEDE\"> though</span><span style=\"background-color: #FFF0F0\"> out</span><span style=\"background-color: #FF0000\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFEFE\"> m</span><span style=\"background-color: #FFFDFD\"> sure</span><span style=\"background-color: #FFF5F5\"> i</span><span style=\"background-color: #FFF6F6\"> will</span><span style=\"background-color: #FFDADA\"> show</span><span style=\"background-color: #FFF7F7\"> it</span><span style=\"background-color: #FFCFCF\"> to</span><span style=\"background-color: #FFF2F2\"> my</span><span style=\"background-color: #FF8383\"> children</span><span style=\"background-color: #FFFAFA\"> in</span><span style=\"background-color: #FFFAFA\"> the</span><span style=\"background-color: #FFF2F2\"> future</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FFFBFB\"> when</span><span style=\"background-color: #FFFAFA\"> i</span><span style=\"background-color: #FFFBFB\"> have</span><span style=\"background-color: #FFE1E1\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_7]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFF6F6\"> watched</span><span style=\"background-color: #FFCBCB\"> this</span><span style=\"background-color: #FFFDFD\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFDFD\"> child</span><span style=\"background-color: #FFF8F8\"> and</span><span style=\"background-color: #FFFAFA\"> still</span><span style=\"background-color: #FFF1F1\"> enjoy</span><span style=\"background-color: #FFF8F8\"> viewing</span><span style=\"background-color: #FFE4E4\"> it</span><span style=\"background-color: #FFF8F8\"> every</span><span style=\"background-color: #FFFBFB\"> once</span><span style=\"background-color: #FFFBFB\"> in</span><span style=\"background-color: #FFF7F7\"> a</span><span style=\"background-color: #FFEBEB\"> while</span><span style=\"background-color: #FFF7F7\"> for</span><span style=\"background-color: #FFF8F8\"> the</span><span style=\"background-color: #FFFBFB\"> nostalgia</span><span style=\"background-color: #FFF7F7\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFDFD\"> was</span><span style=\"background-color: #FFFDFD\"> younger</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FF5E5E\"> loved</span><span style=\"background-color: #FFE4E4\"> the</span><span style=\"background-color: #FFFBFB\"> movie</span><span style=\"background-color: #FFE1E1\"> because</span><span style=\"background-color: #FFA4A4\"> of</span><span style=\"background-color: #FF8080\"> the</span><span style=\"background-color: #FF2C2C\"> entertaining</span><span style=\"background-color: #FFFBFB\"> storyline</span><span style=\"background-color: #FF6868\"> and</span><span style=\"background-color: #FFD2D2\"> interesting</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE5E5\"> today</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFFCFC\"> i</span><span style=\"background-color: #FFEDED\"> still</span><span style=\"background-color: #FF4E4E\"> love</span><span style=\"background-color: #FFEDED\"> the</span><span style=\"background-color: #FFFDFD\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFE5E5\"> additionally</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFFAFA\"> i</span><span style=\"background-color: #FFE8E8\"> think</span><span style=\"background-color: #FFF4F4\"> of</span><span style=\"background-color: #FFEDED\"> the</span><span style=\"background-color: #FFFCFC\"> plot</span><span style=\"background-color: #FFB8B8\"> with</span><span style=\"background-color: #FFCFCF\"> higher</span><span style=\"background-color: #FF5F5F\"> regard</span><span style=\"background-color: #FFE6E6\"> because</span><span style=\"background-color: #FFF6F6\"> i</span><span style=\"background-color: #FFB4B4\"> now</span><span style=\"background-color: #FF8F8F\"> see</span><span style=\"background-color: #FF8B8B\"> the</span><span style=\"background-color: #FFF4F4\"> morals</span><span style=\"background-color: #FF9292\"> and</span><span style=\"background-color: #FFF1F1\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFEFE\"> ##e</span><span style=\"background-color: #FFEFEF\"> is</span><span style=\"background-color: #FF9696\"> far</span><span style=\"background-color: #FFD8D8\"> from</span><span style=\"background-color: #FFE6E6\"> the</span><span style=\"background-color: #FF0000\"> worst</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFF6F6\"> ever</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFF1F1\"> and</span><span style=\"background-color: #FFDBDB\"> though</span><span style=\"background-color: #FFF9F9\"> out</span><span style=\"background-color: #FFF5F5\"> dated</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFCFC\"> m</span><span style=\"background-color: #FFF4F4\"> sure</span><span style=\"background-color: #FFFCFC\"> i</span><span style=\"background-color: #FFFCFC\"> will</span><span style=\"background-color: #FFFCFC\"> show</span><span style=\"background-color: #FFE4E4\"> it</span><span style=\"background-color: #FFFDFD\"> to</span><span style=\"background-color: #FFFDFD\"> my</span><span style=\"background-color: #FFFDFD\"> children</span><span style=\"background-color: #FFFDFD\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> future</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFDFD\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_8]<br><span style=\"background-color: #FFFDFD\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> watched</span><span style=\"background-color: #FFFEFE\"> this</span><span style=\"background-color: #FFFDFD\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFDFD\"> child</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFEFE\"> enjoy</span><span style=\"background-color: #FFFEFE\"> viewing</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFEFE\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFFEFE\"> for</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> nostalgia</span><span style=\"background-color: #FFFEFE\"> factor</span><span style=\"background-color: #FF0606\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> entertaining</span><span style=\"background-color: #FFFEFE\"> storyline</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFDFD\"> characters</span><span style=\"background-color: #FF3434\"> .</span><span style=\"background-color: #FFFCFC\"> today</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFEFE\"> love</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> characters</span><span style=\"background-color: #FF0707\"> .</span><span style=\"background-color: #FFFEFE\"> additionally</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> think</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> plot</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFEFE\"> higher</span><span style=\"background-color: #FFFEFE\"> regard</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> now</span><span style=\"background-color: #FFFEFE\"> see</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> morals</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> symbolism</span><span style=\"background-color: #FF0606\"> .</span><span style=\"background-color: #FFE1E1\"> rainbow</span><span style=\"background-color: #FFEEEE\"> brit</span><span style=\"background-color: #FFFAFA\"> ##e</span><span style=\"background-color: #FFFEFE\"> is</span><span style=\"background-color: #FFFEFE\"> far</span><span style=\"background-color: #FFFEFE\"> from</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> worst</span><span style=\"background-color: #FFFDFD\"> film</span><span style=\"background-color: #FFFEFE\"> ever</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> though</span><span style=\"background-color: #FFFEFE\"> out</span><span style=\"background-color: #FFFEFE\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> m</span><span style=\"background-color: #FFFEFE\"> sure</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> will</span><span style=\"background-color: #FFFEFE\"> show</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFEFE\"> my</span><span style=\"background-color: #FFFBFB\"> children</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> future</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFE7E7\"> children</span><span style=\"background-color: #FF0000\"> .</span><br><br>[BERTのAttentionを可視化_9]<br><span style=\"background-color: #FFF1F1\"> [CLS]</span><span style=\"background-color: #FFEEEE\"> i</span><span style=\"background-color: #FFD8D8\"> watched</span><span style=\"background-color: #FFD4D4\"> this</span><span style=\"background-color: #FFDCDC\"> movie</span><span style=\"background-color: #FFFCFC\"> as</span><span style=\"background-color: #FFFDFD\"> a</span><span style=\"background-color: #FFFBFB\"> child</span><span style=\"background-color: #FFC9C9\"> and</span><span style=\"background-color: #FFE0E0\"> still</span><span style=\"background-color: #FFD4D4\"> enjoy</span><span style=\"background-color: #FFF9F9\"> viewing</span><span style=\"background-color: #FFF5F5\"> it</span><span style=\"background-color: #FFFCFC\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFCFC\"> while</span><span style=\"background-color: #FFF1F1\"> for</span><span style=\"background-color: #FFF5F5\"> the</span><span style=\"background-color: #FFEDED\"> nostalgia</span><span style=\"background-color: #FFF5F5\"> factor</span><span style=\"background-color: #FFF6F6\"> .</span><span style=\"background-color: #FFFDFD\"> when</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFDFD\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFF9F9\"> i</span><span style=\"background-color: #FFE5E5\"> loved</span><span style=\"background-color: #FFEBEB\"> the</span><span style=\"background-color: #FFEAEA\"> movie</span><span style=\"background-color: #FFF2F2\"> because</span><span style=\"background-color: #FFF7F7\"> of</span><span style=\"background-color: #FFCBCB\"> the</span><span style=\"background-color: #FFBDBD\"> entertaining</span><span style=\"background-color: #FFEFEF\"> storyline</span><span style=\"background-color: #FFDCDC\"> and</span><span style=\"background-color: #FFF5F5\"> interesting</span><span style=\"background-color: #FFFDFD\"> characters</span><span style=\"background-color: #FFF7F7\"> .</span><span style=\"background-color: #FFE0E0\"> today</span><span style=\"background-color: #FFF6F6\"> ,</span><span style=\"background-color: #FFE4E4\"> i</span><span style=\"background-color: #FFD5D5\"> still</span><span style=\"background-color: #FF2727\"> love</span><span style=\"background-color: #FFF7F7\"> the</span><span style=\"background-color: #FFF9F9\"> characters</span><span style=\"background-color: #FFF6F6\"> .</span><span style=\"background-color: #FFEDED\"> additionally</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFEDED\"> i</span><span style=\"background-color: #FF9D9D\"> think</span><span style=\"background-color: #FFFBFB\"> of</span><span style=\"background-color: #FFDBDB\"> the</span><span style=\"background-color: #FFD1D1\"> plot</span><span style=\"background-color: #FFEFEF\"> with</span><span style=\"background-color: #FFF3F3\"> higher</span><span style=\"background-color: #FFD9D9\"> regard</span><span style=\"background-color: #FFF6F6\"> because</span><span style=\"background-color: #FFF1F1\"> i</span><span style=\"background-color: #FFD5D5\"> now</span><span style=\"background-color: #FFC3C3\"> see</span><span style=\"background-color: #FF8585\"> the</span><span style=\"background-color: #FFB8B8\"> morals</span><span style=\"background-color: #FFC9C9\"> and</span><span style=\"background-color: #FF9292\"> symbolism</span><span style=\"background-color: #FFF6F6\"> .</span><span style=\"background-color: #FFFDFD\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFAFA\"> ##e</span><span style=\"background-color: #FF4848\"> is</span><span style=\"background-color: #FFB3B3\"> far</span><span style=\"background-color: #FFECEC\"> from</span><span style=\"background-color: #FFC7C7\"> the</span><span style=\"background-color: #FFC7C7\"> worst</span><span style=\"background-color: #FFF1F1\"> film</span><span style=\"background-color: #FFF5F5\"> ever</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFA0A0\"> and</span><span style=\"background-color: #FFF0F0\"> though</span><span style=\"background-color: #FFFCFC\"> out</span><span style=\"background-color: #FFF2F2\"> dated</span><span style=\"background-color: #FFFCFC\"> ,</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFFAFA\"> m</span><span style=\"background-color: #FFCECE\"> sure</span><span style=\"background-color: #FFE3E3\"> i</span><span style=\"background-color: #FFF0F0\"> will</span><span style=\"background-color: #FF5050\"> show</span><span style=\"background-color: #FF0000\"> it</span><span style=\"background-color: #FFF7F7\"> to</span><span style=\"background-color: #FFF9F9\"> my</span><span style=\"background-color: #FFF5F5\"> children</span><span style=\"background-color: #FFFCFC\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFDFD\"> future</span><span style=\"background-color: #FFDEDE\"> ,</span><span style=\"background-color: #FFFAFA\"> when</span><span style=\"background-color: #FFF8F8\"> i</span><span style=\"background-color: #FFEFEF\"> have</span><span style=\"background-color: #FFF2F2\"> children</span><span style=\"background-color: #FFF6F6\"> .</span><br><br>[BERTのAttentionを可視化_10]<br><span style=\"background-color: #FFF2F2\"> [CLS]</span><span style=\"background-color: #FFF4F4\"> i</span><span style=\"background-color: #FFFEFE\"> watched</span><span style=\"background-color: #FFF1F1\"> this</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFDFD\"> child</span><span style=\"background-color: #FFF8F8\"> and</span><span style=\"background-color: #FFF3F3\"> still</span><span style=\"background-color: #FFFBFB\"> enjoy</span><span style=\"background-color: #FFFEFE\"> viewing</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFEFE\"> every</span><span style=\"background-color: #FFFEFE\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> while</span><span style=\"background-color: #FFFBFB\"> for</span><span style=\"background-color: #FFFBFB\"> the</span><span style=\"background-color: #FFFCFC\"> nostalgia</span><span style=\"background-color: #FFFDFD\"> factor</span><span style=\"background-color: #FFFCFC\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFF9F9\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFDFD\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFDFD\"> because</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> entertaining</span><span style=\"background-color: #FFFEFE\"> storyline</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFCFC\"> .</span><span style=\"background-color: #FF0000\"> today</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFD0D0\"> i</span><span style=\"background-color: #FFCDCD\"> still</span><span style=\"background-color: #FFECEC\"> love</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFCFC\"> .</span><span style=\"background-color: #FFC6C6\"> additionally</span><span style=\"background-color: #FFFAFA\"> ,</span><span style=\"background-color: #FFDCDC\"> i</span><span style=\"background-color: #FFF5F5\"> think</span><span style=\"background-color: #FFFBFB\"> of</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFEFE\"> plot</span><span style=\"background-color: #FFC6C6\"> with</span><span style=\"background-color: #FFFBFB\"> higher</span><span style=\"background-color: #FFF0F0\"> regard</span><span style=\"background-color: #FFFAFA\"> because</span><span style=\"background-color: #FFEEEE\"> i</span><span style=\"background-color: #FF8686\"> now</span><span style=\"background-color: #FFFBFB\"> see</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFDFD\"> morals</span><span style=\"background-color: #FFF8F8\"> and</span><span style=\"background-color: #FFFEFE\"> symbolism</span><span style=\"background-color: #FFFCFC\"> .</span><span style=\"background-color: #FFFEFE\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFEFE\"> ##e</span><span style=\"background-color: #FFFEFE\"> is</span><span style=\"background-color: #FFFCFC\"> far</span><span style=\"background-color: #FFFCFC\"> from</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> worst</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFDFD\"> ever</span><span style=\"background-color: #FFFDFD\"> ,</span><span style=\"background-color: #FFEEEE\"> and</span><span style=\"background-color: #FFFEFE\"> though</span><span style=\"background-color: #FFFEFE\"> out</span><span style=\"background-color: #FFFEFE\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFF5F5\"> i</span><span style=\"background-color: #FFFDFD\"> m</span><span style=\"background-color: #FFC2C2\"> sure</span><span style=\"background-color: #FFF5F5\"> i</span><span style=\"background-color: #FFF2F2\"> will</span><span style=\"background-color: #FFFEFE\"> show</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFCFC\"> to</span><span style=\"background-color: #FFF5F5\"> my</span><span style=\"background-color: #FFFDFD\"> children</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFCFC\"> future</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFECEC\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFCFC\"> .</span><br><br>[BERTのAttentionを可視化_11]<br><span style=\"background-color: #FFFEFE\"> [CLS]</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFDFD\"> watched</span><span style=\"background-color: #FFFDFD\"> this</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> as</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFEFE\"> child</span><span style=\"background-color: #FFFEFE\"> and</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFEFE\"> enjoy</span><span style=\"background-color: #FFFEFE\"> viewing</span><span style=\"background-color: #FFFEFE\"> it</span><span style=\"background-color: #FFFCFC\"> every</span><span style=\"background-color: #FFFCFC\"> once</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> a</span><span style=\"background-color: #FFFCFC\"> while</span><span style=\"background-color: #FFFDFD\"> for</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFEFE\"> nostalgia</span><span style=\"background-color: #FFFDFD\"> factor</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> was</span><span style=\"background-color: #FFFEFE\"> younger</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> loved</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> movie</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFDFD\"> of</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFCFC\"> entertaining</span><span style=\"background-color: #FFFEFE\"> storyline</span><span style=\"background-color: #FFFBFB\"> and</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> today</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> still</span><span style=\"background-color: #FFFEFE\"> love</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> additionally</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFDFD\"> think</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFFEFE\"> plot</span><span style=\"background-color: #FFFEFE\"> with</span><span style=\"background-color: #FFFDFD\"> higher</span><span style=\"background-color: #FFFEFE\"> regard</span><span style=\"background-color: #FFFEFE\"> because</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFDFD\"> now</span><span style=\"background-color: #FFF9F9\"> see</span><span style=\"background-color: #FFFAFA\"> the</span><span style=\"background-color: #FFFEFE\"> morals</span><span style=\"background-color: #FFFCFC\"> and</span><span style=\"background-color: #FFFEFE\"> symbolism</span><span style=\"background-color: #FFFEFE\"> .</span><span style=\"background-color: #FFFEFE\"> rainbow</span><span style=\"background-color: #FFFEFE\"> brit</span><span style=\"background-color: #FFFEFE\"> ##e</span><span style=\"background-color: #FFFEFE\"> is</span><span style=\"background-color: #FFFDFD\"> far</span><span style=\"background-color: #FFFEFE\"> from</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FF0000\"> worst</span><span style=\"background-color: #FFFEFE\"> film</span><span style=\"background-color: #FFFEFE\"> ever</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFCFC\"> and</span><span style=\"background-color: #FFFEFE\"> though</span><span style=\"background-color: #FFFEFE\"> out</span><span style=\"background-color: #FFFEFE\"> dated</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> m</span><span style=\"background-color: #FFFEFE\"> sure</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> will</span><span style=\"background-color: #FFFCFC\"> show</span><span style=\"background-color: #FFFDFD\"> it</span><span style=\"background-color: #FFFEFE\"> to</span><span style=\"background-color: #FFFEFE\"> my</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> in</span><span style=\"background-color: #FFFEFE\"> the</span><span style=\"background-color: #FFFEFE\"> future</span><span style=\"background-color: #FFFEFE\"> ,</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFFEFE\"> i</span><span style=\"background-color: #FFFEFE\"> have</span><span style=\"background-color: #FFFEFE\"> children</span><span style=\"background-color: #FFFEFE\"> .</span><br><br>[BERTのAttentionを可視化_12]<br><span style=\"background-color: #FFFDFD\"> [CLS]</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFF8F8\"> watched</span><span style=\"background-color: #FFE4E4\"> this</span><span style=\"background-color: #FFF9F9\"> movie</span><span style=\"background-color: #FFFCFC\"> as</span><span style=\"background-color: #FFFBFB\"> a</span><span style=\"background-color: #FFF9F9\"> child</span><span style=\"background-color: #FFF9F9\"> and</span><span style=\"background-color: #FFFDFD\"> still</span><span style=\"background-color: #FFFCFC\"> enjoy</span><span style=\"background-color: #FFFEFE\"> viewing</span><span style=\"background-color: #FFFDFD\"> it</span><span style=\"background-color: #FFFBFB\"> every</span><span style=\"background-color: #FFFDFD\"> once</span><span style=\"background-color: #FFFAFA\"> in</span><span style=\"background-color: #FFFAFA\"> a</span><span style=\"background-color: #FFFCFC\"> while</span><span style=\"background-color: #FFFCFC\"> for</span><span style=\"background-color: #FFF9F9\"> the</span><span style=\"background-color: #FFF4F4\"> nostalgia</span><span style=\"background-color: #FFFBFB\"> factor</span><span style=\"background-color: #FFF9F9\"> .</span><span style=\"background-color: #FFFEFE\"> when</span><span style=\"background-color: #FFF5F5\"> i</span><span style=\"background-color: #FFFDFD\"> was</span><span style=\"background-color: #FFFAFA\"> younger</span><span style=\"background-color: #FFFBFB\"> i</span><span style=\"background-color: #FFFAFA\"> loved</span><span style=\"background-color: #FFFAFA\"> the</span><span style=\"background-color: #FFFBFB\"> movie</span><span style=\"background-color: #FFFDFD\"> because</span><span style=\"background-color: #FFFEFE\"> of</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFDFD\"> entertaining</span><span style=\"background-color: #FFFEFE\"> storyline</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFEFE\"> interesting</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFFAFA\"> .</span><span style=\"background-color: #FFF9F9\"> today</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FFF3F3\"> i</span><span style=\"background-color: #FFFCFC\"> still</span><span style=\"background-color: #FFF4F4\"> love</span><span style=\"background-color: #FFFDFD\"> the</span><span style=\"background-color: #FFFEFE\"> characters</span><span style=\"background-color: #FFF9F9\"> .</span><span style=\"background-color: #FFF4F4\"> additionally</span><span style=\"background-color: #FFFBFB\"> ,</span><span style=\"background-color: #FFF6F6\"> i</span><span style=\"background-color: #FFFAFA\"> think</span><span style=\"background-color: #FFF7F7\"> of</span><span style=\"background-color: #FFEFEF\"> the</span><span style=\"background-color: #FFFBFB\"> plot</span><span style=\"background-color: #FFFBFB\"> with</span><span style=\"background-color: #FFF6F6\"> higher</span><span style=\"background-color: #FFF9F9\"> regard</span><span style=\"background-color: #FFFAFA\"> because</span><span style=\"background-color: #FFF9F9\"> i</span><span style=\"background-color: #FFFBFB\"> now</span><span style=\"background-color: #FFFAFA\"> see</span><span style=\"background-color: #FFFBFB\"> the</span><span style=\"background-color: #FFFDFD\"> morals</span><span style=\"background-color: #FFFDFD\"> and</span><span style=\"background-color: #FFFEFE\"> symbolism</span><span style=\"background-color: #FFF9F9\"> .</span><span style=\"background-color: #FFFCFC\"> rainbow</span><span style=\"background-color: #FFFDFD\"> brit</span><span style=\"background-color: #FFFBFB\"> ##e</span><span style=\"background-color: #FFEFEF\"> is</span><span style=\"background-color: #FFF7F7\"> far</span><span style=\"background-color: #FFF9F9\"> from</span><span style=\"background-color: #FFE0E0\"> the</span><span style=\"background-color: #FF0000\"> worst</span><span style=\"background-color: #FFF4F4\"> film</span><span style=\"background-color: #FFF4F4\"> ever</span><span style=\"background-color: #FFF8F8\"> ,</span><span style=\"background-color: #FFEFEF\"> and</span><span style=\"background-color: #FFFDFD\"> though</span><span style=\"background-color: #FFF9F9\"> out</span><span style=\"background-color: #FFDFDF\"> dated</span><span style=\"background-color: #FFFAFA\"> ,</span><span style=\"background-color: #FFF7F7\"> i</span><span style=\"background-color: #FFFDFD\"> m</span><span style=\"background-color: #FFF0F0\"> sure</span><span style=\"background-color: #FFF4F4\"> i</span><span style=\"background-color: #FFFBFB\"> will</span><span style=\"background-color: #FFECEC\"> show</span><span style=\"background-color: #FFEEEE\"> it</span><span style=\"background-color: #FFFCFC\"> to</span><span style=\"background-color: #FFF0F0\"> my</span><span style=\"background-color: #FFF9F9\"> children</span><span style=\"background-color: #FFFBFB\"> in</span><span style=\"background-color: #FFFCFC\"> the</span><span style=\"background-color: #FFFBFB\"> future</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFFCFC\"> when</span><span style=\"background-color: #FFDBDB\"> i</span><span style=\"background-color: #FFF9F9\"> have</span><span style=\"background-color: #FFF2F2\"> children</span><span style=\"background-color: #FFF9F9\"> .</span><br><br>[BERTのAttentionを可視化_ALL]<br><span style=\"background-color: #FFEBEB\"> [CLS]</span><span style=\"background-color: #FFF3F3\"> i</span><span style=\"background-color: #FFE6E6\"> watched</span><span style=\"background-color: #FFBBBB\"> this</span><span style=\"background-color: #FFCCCC\"> movie</span><span style=\"background-color: #FFFAFA\"> as</span><span style=\"background-color: #FFFAFA\"> a</span><span style=\"background-color: #FFF9F9\"> child</span><span style=\"background-color: #FFDDDD\"> and</span><span style=\"background-color: #FFE7E7\"> still</span><span style=\"background-color: #FFE7E7\"> enjoy</span><span style=\"background-color: #FFD3D3\"> viewing</span><span style=\"background-color: #FFDADA\"> it</span><span style=\"background-color: #FFD9D9\"> every</span><span style=\"background-color: #FFF8F8\"> once</span><span style=\"background-color: #FFFAFA\"> in</span><span style=\"background-color: #FFF9F9\"> a</span><span style=\"background-color: #FFF2F2\"> while</span><span style=\"background-color: #FFE3E3\"> for</span><span style=\"background-color: #FFC0C0\"> the</span><span style=\"background-color: #FFCDCD\"> nostalgia</span><span style=\"background-color: #FFC8C8\"> factor</span><span style=\"background-color: #FFA7A7\"> .</span><span style=\"background-color: #FFFDFD\"> when</span><span style=\"background-color: #FFF8F8\"> i</span><span style=\"background-color: #FFFDFD\"> was</span><span style=\"background-color: #FFFCFC\"> younger</span><span style=\"background-color: #FFFAFA\"> i</span><span style=\"background-color: #FFD4D4\"> loved</span><span style=\"background-color: #FFDDDD\"> the</span><span style=\"background-color: #FFE2E2\"> movie</span><span style=\"background-color: #FFEFEF\"> because</span><span style=\"background-color: #FFE2E2\"> of</span><span style=\"background-color: #FF9D9D\"> the</span><span style=\"background-color: #FFB1B1\"> entertaining</span><span style=\"background-color: #FFDFDF\"> storyline</span><span style=\"background-color: #FFCACA\"> and</span><span style=\"background-color: #FFE4E4\"> interesting</span><span style=\"background-color: #FFF0F0\"> characters</span><span style=\"background-color: #FFB4B4\"> .</span><span style=\"background-color: #FF9191\"> today</span><span style=\"background-color: #FFF7F7\"> ,</span><span style=\"background-color: #FFDEDE\"> i</span><span style=\"background-color: #FFDEDE\"> still</span><span style=\"background-color: #FF8383\"> love</span><span style=\"background-color: #FFE5E5\"> the</span><span style=\"background-color: #FFF0F0\"> characters</span><span style=\"background-color: #FFA7A7\"> .</span><span style=\"background-color: #FFDDDD\"> additionally</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFE3E3\"> i</span><span style=\"background-color: #FFAAAA\"> think</span><span style=\"background-color: #FFECEC\"> of</span><span style=\"background-color: #FF8383\"> the</span><span style=\"background-color: #FFDBDB\"> plot</span><span style=\"background-color: #FFD8D8\"> with</span><span style=\"background-color: #FFECEC\"> higher</span><span style=\"background-color: #FFBFBF\"> regard</span><span style=\"background-color: #FFE7E7\"> because</span><span style=\"background-color: #FFECEC\"> i</span><span style=\"background-color: #FFC0C0\"> now</span><span style=\"background-color: #FFC1C1\"> see</span><span style=\"background-color: #FF8383\"> the</span><span style=\"background-color: #FFCACA\"> morals</span><span style=\"background-color: #FFC9C9\"> and</span><span style=\"background-color: #FFCECE\"> symbolism</span><span style=\"background-color: #FFA7A7\"> .</span><span style=\"background-color: #FFF1F1\"> rainbow</span><span style=\"background-color: #FFF7F7\"> brit</span><span style=\"background-color: #FFE5E5\"> ##e</span><span style=\"background-color: #FF5757\"> is</span><span style=\"background-color: #FFD1D1\"> far</span><span style=\"background-color: #FFD2D2\"> from</span><span style=\"background-color: #FF8E8E\"> the</span><span style=\"background-color: #FF0000\"> worst</span><span style=\"background-color: #FFAEAE\"> film</span><span style=\"background-color: #FFF1F1\"> ever</span><span style=\"background-color: #FFF5F5\"> ,</span><span style=\"background-color: #FFAAAA\"> and</span><span style=\"background-color: #FFE6E6\"> though</span><span style=\"background-color: #FFF2F2\"> out</span><span style=\"background-color: #FFACAC\"> dated</span><span style=\"background-color: #FFF9F9\"> ,</span><span style=\"background-color: #FFEDED\"> i</span><span style=\"background-color: #FFF8F8\"> m</span><span style=\"background-color: #FFDCDC\"> sure</span><span style=\"background-color: #FFE2E2\"> i</span><span style=\"background-color: #FFF2F2\"> will</span><span style=\"background-color: #FF9393\"> show</span><span style=\"background-color: #FF7777\"> it</span><span style=\"background-color: #FFECEC\"> to</span><span style=\"background-color: #FFEBEB\"> my</span><span style=\"background-color: #FFDCDC\"> children</span><span style=\"background-color: #FFFAFA\"> in</span><span style=\"background-color: #FFF6F6\"> the</span><span style=\"background-color: #FFF8F8\"> future</span><span style=\"background-color: #FFECEC\"> ,</span><span style=\"background-color: #FFFAFA\"> when</span><span style=\"background-color: #FFD4D4\"> i</span><span style=\"background-color: #FFF4F4\"> have</span><span style=\"background-color: #FFE5E5\"> children</span><span style=\"background-color: #FFA5A5\"> .</span><br><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = 61  # 出力させたいデータ\n",
    "html_output = mk_html(index, batch, preds, attention_probs, TEXT)  # HTML作成\n",
    "HTML(html_output)  # HTML形式で出力\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
